Scanning the headlines over the past year, you might be forgiven for believing that in-the-air gestures were coming soon to living room boxes, PCs, laptops, tablets and smartphones.
But in-the-air gestures won’t be a mainstream user interface any time soon. I’ll tell you exactly why at the end of this column. But first, let’s face the fail.
Amazon and Microsoft are both rumored to be working on smartphones that have front-facing optical sensors similar to the technology in Microsoft’s Kinect lines of products.
Kinect blew people away when Microsoft’s Kinect for Xbox 360 first hit the market. The technology enabled a certain kind of game where the system knew your position and movements. It responded to in-the-air gestures.
So with similar technology, you’d think that these phones would also support in-the-air gestures.
Amazon’s phone will probably be announced June 18 at a scheduled event. The phone is expected to use four front-facing sensors to recognize the orientation and location of the face that’s looking at the screen so the screen can show 3D images without goofy glasses.
Microsoft, meanwhile, is reportedly working on a future phone under its mobile devices division purchased from Nokia that has “Kinect-like features” such as sensors that enable gestures. (The report is based on anonymous sources.)
At first, that sounds like Kinect built into a phone, leading to in-the-air gestures, gaming and all the rest.
In fact, most of the gestures are handy but limited. For example, the report says that leading gesture contenders are that you can answer calls by holding the phone to your ear; put the phone on speaker by placing it on a table; mute calls by holding your hand over the phone; and wake up the phone by picking it up.
These “gestures” probably combine visual sensors and motion sensors in the phone. Only one true in-the-air gesture was suggested in the report: waving your hand in front of the screen to dismiss an alert.
So while both of these phones are likely to contain the sensors for in-the-air gestures, they won’t really do much to take advantage of that interface. The companies making these phones know that in-the-air gesture control is a losing proposition, even when the sensors are going to be built in anyway.
When Leap Motion‘s motion controller product was in beta and being demonstrated at high-visibility events, everyone Ooh’d and Aah’d at the potential for in-the-air gesture control for everything. Pundits predicted a bold new interface would soon take over.
But one year after the product actually shipped, I don’t see people using it.
Yes, startups, science projects, makers and tinkerers are getting a lot out of the Leap Motion controller. But it’s nonexistent as a mainstream user interface. The public is rejecting the interface.
By the way, HP will soon ship a PC keyboard with a Leap Motion controller built in, and I’m predicting it won’t be a hot seller.
In-the-Air Gestures Losing Out for Games
Kinect for Xbox was the crown jewel of in-the-air gesture control. When the Xbox One shipped less than a year ago, Kinect was a required component of that gaming system. But the console wasn’t competing against the Sony PlayStation as Microsoft had hoped.
Why In-the-Air Gestures Are Failing as a Mainstream User Interface
So they fixed the problem by offering a Kinect-free Xbox version at a lower price of around $400. At this week’s E3, Kinect was barely mentioned by Microsoft, and some tech publications are calling it “dead in the water.”
Xbox is a platform. And with Kinect and in-the-air gestures on the ropes, developers may shun the interface, leading to a snowball effect that results in the termination of Kinect.
Google recently bought a company called Nest, best known for a smart thermostat product. Nest had more recently launched a smart smoke detector called the Nest Protect Smoke + CO alarm, which offers in-the-air gesture control. If an alarm went off because, let’s say you burnt the toast, you could silence the smoke alarm with a simple wave of your hand.
Unfortunately, the company had to recall some 440,000 units because the gesture feature didn’t work right. Specifically, it was too easy to silence the alarm in a real fire; in other words, the gesture interface gave too many “false positives.” Nest fixed the problem.
Despite the initial glitch and recall, the use of in-the-air gestures actually makes sense for a smoke detector for the practical reality that they’re usually too high to reach. It makes sense because in-the-air gesture control is better than fetching a ladder.
Why In-the-Air Gesture Control Is Failing
Controlling smart devices with in-the-air gestures is a cool idea. And some of the technology is amazing.
The problem is that in-the-air gestures are often applied as an abstraction interface, like the mouse. What I mean by that is that with a mouse you move the pointer to a certain place to effect change on the screen over here. You move one thing in order to move another.
In-the-air gesture interfaces are the same thing. You wave your hands in front of the screen in order to make some change on the screen.
And such abstract user interfaces can’t compete with touch interfaces, like the ones we have on our phones, tablets and some PCs. The reason is that our brains are hardwired for direct manipulation of objects in our environment. When we want to move something, we want to touch and move the thing itself, not something virtual.
In other words, in-the-air gestures for control present too great a “leap” (pun intended) for the human animal.
People do use gestures, but never to control or manipulate things. We use gestures to communicate. And that’s where in-the-air gestures will shine.
Just as we use natural hand gestures while talking, we can expect that using in-the-air gestures for communicating with computers—for computers to understand us better when we’re interacting with them—will be the place where our current in-the-air gestures technology will finally take off and go mainstream.
When computers understand a shrug to mean “I don’t know” and understand a thumbs-up to mean “yes” or “I like it,” then in-the-air gestures will find mainstream acceptance.
But until in-the-air gesture technology is applied to communication, rather than control, we can safely wave off in-the-air gestures as a gimmick, or an interface for obscure outliers or another layer on the ash heap of cool technology rejected by the public.