Nvidia’s new AI-powered platform, Signs, teaches users American Sign Language by tracking learners’ movements through their webcams and providing real-time feedback. The tool currently has a library of 100 signs validated by fluent ASL users and interpreters, but its dataset is being expanded. Created with the American Society for Deaf Children (ASDC) and creative agency DEPT, Signs demonstrates gestures with a 3D avatar, simultaneously records the user practicing them, and analyzes the footage using AI to offer tips such as “wait” and “adjust hand rotation.”
“Most deaf children are born to hearing parents,” ASDC Executive Director Cheri Dowling said in a press release. “Giving family members accessible tools like Signs to start learning ASL early enables them to open an effective communication channel with children as young as six to eight months old.”
Dowling noted that the web platform makes it easier for families with deaf children to continue their ASL education, as it removes the need for a formal classroom and lets parents merge learning with play with their kids. But Signs is not quite in its finished state yet. Nvidia aims to grow the number of validated signs from 100 to 1,000 by collecting 400,000 video clips of demonstrations. Signers of all skill levels can choose the “Record Signs” option on the Signs website and contribute to the dataset.
Image: Nvidia
Signs points to the future with new features
In its current form, Signs focuses on hand movements and finger position, but ASL actually incorporates facial expressions and head movements, too. Researchers plan to use the video library to integrate these additional elements into the learning platform, as well as other nuances like slang terms and regional variations.
In addition, Nvidia wants to use the video library to develop applications other than Signs which “break down communication barriers between the deaf and hearing communities.” The company intends to make the library open source later this year so that other companies can use it to build accessible technologies, like AI agents and video conferencing tools.
Nvidia says that there are vastly fewer AI tools developed with ASL data than English and Spanish, despite them making up the U.S.’s top three most common languages. Nevertheless, in November, Mozilla released free, high-quality voice training data in 180 languages, creating one of the largest open AI voice datasets and helping to improve inclusivity in AI tools.