Columbia Engineering Teaches Robot to Learn Human Lip Movements

Robots have learned to walk, gesture, and talk, but their faces have always lagged behind. Now, a new robot has taken a major step past stiff, unnatural expressions by learning how to move its lips in sync with speech.

The advance targets one of the most stubborn problems in humanoid robotics: facial movement that looks convincing in real conversation, not just in demos. Researchers say even small gains in lip realism can dramatically change how people perceive and respond to a robot.

Crossing the ‘uncanny valley’

Despite advances in humanoid robotics, realistic facial movement, especially around the mouth, has remained difficult to achieve. Most robots still rely on predefined facial motions tied to audio, an approach Columbia Engineering researchers say often produces speech that looks technically correct but feels off in person.

“We may forgive a funny walking gait or an awkward hand motion, but we remain unforgiving of even the slightest facial malgesture,” said Hod Lipson, director of Columbia’s Creative Machines Lab. He described that unforgiving standard as the “uncanny valley,” the point where near-human robots begin to feel lifeless or creepy to people.

According to researchers at Columbia Engineering, the robot, called EMO, wasn’t programmed with rigid facial rules; instead, it learned lip motion by watching. First, by studying its own reflection, then by observing how humans speak and sing, allowing it to mimic realistic lip movements in real time.

From speech to song

In demonstrations, the robot’s mouth moved in time with spoken audio rather than lagging behind it or snapping between fixed shapes. As words played, its lips formed rounded vowels, tight closures, and subtle transitions that closely tracked the rhythm of speech, making it feel deliberate, not mechanical.

That responsiveness carried across different languages and vocal styles. The robot was shown lip-syncing phrases spoken in multiple languages, adjusting its mouth shape to unfamiliar sounds without needing language-specific tuning. Researchers noted that it does this without understanding the words themselves, responding purely to what it hears.

The effect became most noticeable when the robot sang. In one test, it performed an AI-generated song from its debut album, titled “hello world,” keeping pace with changes in pitch and tempo while maintaining continuous, expressive lip motion.

The missing piece in human-facing robots

For the Columbia team, realistic lip movement is a requirement for robots meant to operate around people. 

“Much of humanoid robotics today is focused on leg and hand motion, for activities like walking and grasping,” said Lipson. “But facial affection is equally important for any robotic application involving human interaction.” He cited settings like education, healthcare, and elder care, where people naturally rely on facial cues during conversation.

The team said expressive faces are a missing channel in human-robot communication, as they carry emotion, intent, and timing alongside speech. Without it, even advanced robots can feel distant or mechanical. 

For robots designed to work closely with people, the researchers emphasized that facial movement is as critical to interaction as movement or speech.

A robotics startup backed by SoftBank just raised $1.4 billion to build a shared AI “brain” for machines.