eWEEK content and product recommendations are editorially independent. We may make money when you click on links to our partners. Learn More.
While it initially looked like a rout, the humans clawed their way back during the first round of Jeopardy! and forced IBM’s supercomputer to tie for the lead.
The first segment of the long-anticipated two-game Jeorpardy tournament between former Jeopardy! champions, Ken Jennings and Brad Rutter, and Watson aired on Feb. 14. Watson and Rutter both had $5,000 on the board and Jennings trailed behind with $2,000.
“I had a good feeling at the end of the first show,” said IBM’s David Ferrucci, the lead researcher and principal investigator on the Watson project in a post-game analysis posted on the company’s A Smarter Planet blog. “I thought: Everybody will realize the computer is competitive,” he said.
The first day’s final scores revealed only part of the story. Rutter made the first selection, and beat Watson to the buzzer to answer the question. After that, Watson dominated the round up to the commercial break, buzzing in with 11 correct answers out of 15 questions, including the Daily Double (it wagered $1,000 for Literary Character APB). The second half of the round started with $5,200 for Watson, $1,000 for Rutter, and $200 for Jennings on the scoreboard, but Jennings and Rutter beat Watson to the buzzer several times during the course of the round. Most of the wrong answers during this round were also Watson’s.
Viewers who expected the computer to get every question right were treated to several of Watson’s wrong answers in the second half. Unlike a human player, Watson can’t adjust its answers to what other players say and answers whatever it picked as its top answer during initial processing. After Jennings incorrectly answered “20s” was the decade in which Oreo cookies were introduced, Watson answered with “1920s.”
“Watson is very bright, very fast, but he has some weird little moments,” Trebek said.
Another Watson mistake illustrated the challenges of natural language processing. The category was Olympic Oddities, and the answer was a gymnast with an unusual physical feature. Watson answered “leg,” but was ruled incorrect, because the proper answer (edited: no one got right) was that the gymnast’s leg was missing.
In this case, Watson very likely didn’t understand what an “oddity” is, according to Ferrucci’s post-game analysis on Smarter Planet.
“The computer wouldn’t know that a missing leg is odder than anything else,” said Ferrucci.
Watson could come to understand what an odditiy is over time, “by reading more material and playing more games,” according to A Smarter Planet.
Jeopardy requires an enormously broad domain of knowledge with “infinite ways” to present the question, Ferrucci has said in the past. Even if Watson knew a topic well, it still needed to understand what the clue was asking for, he said. The categories of the first round were: Literary Characters APB, Beatles People, Olympic Oddities, Name the Decade, Final Frontiers, and Alternate Meanings. The literary characters category was heavy on puns and wordplay, and while Watson was supremely comfortable in the Beatles category, it got every single question in the decades category wrong.
Watson has an “Achilles heel,” because its betting is restricted to its confidence level, according to “Final Jeopardy,” a book about Watson by technology journalist Stephen Baker. Despite previous reports, Watson actually lost the practice match in January to Jennings in Final Jeopardy, because Jennings had bet aggressively on the Daily Doubles and Final Jeopardy, Baker said in an excerpt to final chapter of his book. The remainder of the chapter will be posted after the final game, and the hardcover of the book will be available on Thursday.
Watsons Practical Applications in Health Care
The IBM team has been working for four years to get to this point, but the company has some ideas of how to turn Watson into practical applications in areas such as health care, tech support and business analytics, Ferrucci said. The system can listen to patients describe their ailments and co-relate it against all the medical journals and relevant information, to come up with a list of potential diagnoses, Katharine Frase, vice-president of industry solutions and emerging business at IBM Research, told eWEEK.
Eight universities collaborated with IBM Research to “advance” the Deep Q&A technology powering Watson, according to IBM. The institutions include Carnegie Mellon University, Massachusetts Institute of Technology, University of Texas at Austin, University of Southern California, Rensselaer Polytechnic Institute, University at Albany, University of Trento in Italy and University of Massachusetts Amherst. Each team will be focused on different aspects, such as the MIT team’s work that allows Watson to break down the question into parts and process concurrently, and the CMU team’s work on the computer’s answer-scoring algorithm, IBM said.
Only the first round was broadcast, with the remaining Double Jeopardy and Final Jeopardy scheduled for Tuesday. The final game will air on Wednesday. The tournament was taped at the IBM research center in Yorktown Heights, N.Y. in January, but Jeopardy and IBM has managed to prevent the final outcome from leaking. The winner will take home the grand prize of $1 million (and bragging rights) which IBM has said will donate to charity if Watson wins.
As part of the players’ introductions, Trebek walked the TV audience through the server room “next door” to the studio to show the five racks containing ten Power7 servers that house Watson’s code, algorithms, and database containing everything it knows. The deep analytics system is the equivalent of 2,800 powerful computers tied together in a super high-speed network, Trebek said.
“It can play with champions. It answered some tough questions,” said Ferrucci.
During the chit-chat segment halfway through round 1, the audience was shown a documentary clip talking about how bad Watson was at answering Jeopardy questions in the early years of development.