eWEEK content and product recommendations are editorially independent. We may make money when you click on links to our partners. Learn More.
As an Xbox peripheral, Kinect may be (mostly) fun and games for Microsoft, but the company’s research arm has other, more serious designs for the sensor technology.
Microsoft Research Asia, working with the Chinese Academy of Sciences and Beijing Union University, has devised a prototype that translates spoken language into sign language, and vice versa, in real time. The project, called Kinect Sign Language Translator, would make it possible for users to translate between different spoken and sign languages.
For instance, the technology can translate an English-speaking person’s speech to Chinese sign language via an on-screen avatar and can potentially be extended to more languages. “During the first six months, we focused mainly on Chinese sign language data collection and labeling,” noted Guobin Wu, research program manager for Microsoft Research Asia, in a Microsoft Research Connections Blog post.
Should the team’s efforts reach commercialization, it could have a big effect on the world’s deaf population. “There are more than 20 million people in China who are hard of hearing, and an estimated 360 million such people around the world, so this project has immense potential to generate positive social impact worldwide,” stated Wu.
An accompanying video shows the device in action. Users stand in front of a Kinect sensor bar, speak or sign their message, and a moment later the system delivers an on-screen translation with audio and an avatar that reproduces the signed speech.
The group drew from several disciplines while trying to develop Kinect Sign Language Translator, said Wu, including “language modeling, translation, computer vision, speech recognition, 3D modeling, and special education.” Yet there is still plenty of work ahead, he cautioned.
While the technology in its current state “can successfully produce good results for translator mode,” which picks up isolated words signed by a specific person, “we are diligently working to overcome the technology hurdles so that the system can reliably understand and interpret in communication mode,” Wu said. The goal of communication mode is to understand “continuous communication from any competent signer.”
Wu added that “while we’re solving those challenges, we are also starting to build up the system’s vocabulary of American Sign Language gestures, which are different from those of Chinese Sign Language.” Nonetheless, Wu said the project is off to a good start and that his group expects “that more and more researchers from different disciplines and different countries will collaboratively build on the prototype, so that the Kinect Sign Language Translator system will ultimately benefit the global community of those who are deaf or hard of hearing.”
Kinect, the popular peripheral for Xbox 360 and now a standard feature on the upcoming Xbox One gaming and media console, has been embraced by the hacking community. Expanding on its motion control capabilities, tech enthusiasts have used the device as a basis for robot guidance systems and advanced 3D imaging systems. In March, the company released Kinect for Windows SDK 1.7, which included support for GPU-assisted 3D scanning.