Hello Monday creates a digital experience to learn American sign language all via webcam

Every year in the United States two to three out of every 1000 children born are deaf or hard-of-hearing. The majority of these children (90 per cent) are also born to hearing parents, “and in many cases, their child is the first deaf person these parents have ever encountered,” explains Anders Jessen, a tech partner at creative studio Hello Monday, whose latest project creates digital opportunities to learn sign language by utilising AI technology.

As Anders continues, if a hard-of-hearing child isn’t introduced to sign language at an early age: “This can lead to language delay or deprivation, which has long-term negative impacts on a child’s life,” he explains. “That is why it is so important that parents of deaf children have the opportunity to learn American Sign Language (ASL) as soon as possible.” Hello Monday’s new project, Fingerspelling, aims to create this opportunity.

An in-browser website which aids the viewer in learning sign language via their webcam and hand-tracking, Fingerspelling is “a bit of an atypical project” for Hello Monday. As a creative studio who loves to experiment with technology, the seed of the idea was actually planted during a research and ideation session. Starting with the idea of potentially marrying hand-tracking with ASL fingerspelling, Anders and his team began “to dive deeper into the world of fingerspelling and the need for better solutions,” he tells It’s Nice That. This began with a “pretty advanced prototype” the studio could share with the American Association for Deaf Children, as a possible partner. “And from there we made it into a real experience.”

Working closely with an ASL professor to ensure all signing is correct on the platform, Fingerspelling tracks the hand movements of a user via MediaPipe’s Hands, a high-fidelity hand and finger tracking solution which “employs machine learning to infer 21 3D landmarks of a hand from just a single frame,” as its website describes. As a studio who often works with a lot of open source AI technologies, the team at Hello Monday were “blown away at both its speed and precision, even for when your fingers are particularly occluded,” adds Anders. “We immediately started to experiment with it to see how we could use this in a meaningful, creative way.”

This comes to life on the platform’s home, Fingerspelling.xyz. At first the website will show you a series of words with a 3D model of how your hand should be positioned to create each letter. Then, as you begin to sign the word yourself via your webcam, the camera will track movements and also offer feedback. “As an example, you would be given the word ‘Able’ – and asked to spell it,” describes Anders. “You are then prompted to sign the first letter, A. On the left-hand side you have a 3D hand that shows you how your hand and fingers should be positioned. On the right hand side you see a feed from your web camera, and then you hold up your hands and mimic what the 3D hand shows an A looks like. There are markers on the 3D hand as well that help you, in real time, to see if your fingers are positioned correctly. And then you move on to the next letter.”

As a digital experience that utilises such advanced machine learning technologies, one of Fingerspelling’s most successful qualities is just how simple it is to use as a viewer. Rather than being an overly complicated digital experience with several hurdles, Hello Monday’s approach feels fully built with the ease of the viewer in mind. There are well-considered design details too, such as the use of Labil Grotesk (the same typeface we use here at It’s Nice That). Considering that “angles and placement of fingers are of key importance when communicating speech through sign language,” Hello Monday were keen to feature a typeface “that reflects similar subtle visual nuances,” says Anders. “Labil Grotesk’s angled glyphs and “dynamic movement” therefore mirrors these gestures in its “slightly off-kilter appearance”, which also feels “joyful and engaging” overall supporting Hello Monday’s visual goals for Fingerspelling.

Now an up-and-running experience to explore and learn from, Hello Monday hopes first and foremost that Fingerspelling aids the “broader public to know and understand that ASL fingerspelling exists and what it is,” explains Anders. Continuing the studio’s goal to “help bridge the communication barrier between the deaf and the hearing”, Hello Monday are currently collecting feedback from the website’s visitors – and a renowned US university who are currently running tests on the platform – to inform an updated version in future. You can test Fingerspelling.xyz directly yourself, here.