Google Exec Outlines Advances in Deep Learning
Dean said getting those relationships right and understanding context are important. When a neural network is talking about a car, "you want to get the sense of 'car-ness,' not that it's spelled C-A-R," he said. Google is researching these and other issues, and there have been some significant gains, he said. By way of example, he noted that in 2011, the best neural network at the annual ImageNet competition had an error rate of 25.7 percent. By 2013, the rate had dropped to 11.7 percent and in 2014, the rate was 6.7 percent. In 2015, the Chinese Web services provider Baidu said it had reached an error rate of 6 percent. Earlier this year, Microsoft published a paper saying it had hit a 4.9 percent error rate. Earlier this month, Google published its own paper outlining an error rate of 4.8 percent.He also noted projects that were run using classic Atari games—such as Space Invaders—to determine how well neural networks learn without any pre-programmed data in front of them. When a neural network was set up to play the game, the initial results were as expected -- the network was beaten pretty quickly. However, it learned each time it played the game, and after several hundred times, the neural network was essentially unbeatable. "Eventually, it just can't get killed," Dean said. "It becomes superhuman." The networks in many ways learn as people do, by what they experience and by recognizing errors and correcting them. However, Dean said that while neural networks are inspired by the human brain, they are not made to mimic the brain. For example, he noted that when the brain tackles a problem, it only "fires" 10 times—essentially the thought process goes through only 10 different layers of highly parallel processing. Neural networks often can have many times more layers of processing.
"This is indicative of the kind of progress we're making," Dean said.