IBM's Watson Ties for Lead on Jeopardy but Makes Some Doozies

In the first broadcast of the much-hyped Jeopardy! battle of man vs. machine, Brad Rutter tied with Watson for first with Ken Jennings not so far behind.

While it initially looked like a rout, the humans clawed their way back during the first round of Jeopardy! and forced IBM's supercomputer to tie for the lead.

The first segment of the long-anticipated two-game Jeorpardy tournament between former Jeopardy! champions, Ken Jennings and Brad Rutter, and Watson aired on Feb. 14. Watson and Rutter both had $5,000 on the board and Jennings trailed behind with $2,000.

"I had a good feeling at the end of the first show," said IBM's David Ferrucci, the lead researcher and principal investigator on the Watson project in a post-game analysis posted on the company's A Smarter Planet blog. "I thought: Everybody will realize the computer is competitive," he said.

The first day's final scores revealed only part of the story. Rutter made the first selection, and beat Watson to the buzzer to answer the question. After that, Watson dominated the round up to the commercial break, buzzing in with 11 correct answers out of 15 questions, including the Daily Double (it wagered $1,000 for Literary Character APB). The second half of the round started with $5,200 for Watson, $1,000 for Rutter, and $200 for Jennings on the scoreboard, but Jennings and Rutter beat Watson to the buzzer several times during the course of the round. Most of the wrong answers during this round were also Watson's.

Viewers who expected the computer to get every question right were treated to several of Watson's wrong answers in the second half. Unlike a human player, Watson can't adjust its answers to what other players say and answers whatever it picked as its top answer during initial processing. After Jennings incorrectly answered "20s" was the decade in which Oreo cookies were introduced, Watson answered with "1920s."

"Watson is very bright, very fast, but he has some weird little moments," Trebek said.

Another Watson mistake illustrated the challenges of natural language processing. The category was Olympic Oddities, and the answer was a gymnast with an unusual physical feature. Watson answered "leg," but was ruled incorrect, because the proper answer (edited: no one got right) was that the gymnast's leg was missing.

In this case, Watson very likely didn't understand what an "oddity" is, according to Ferrucci's post-game analysis on Smarter Planet.

"The computer wouldn't know that a missing leg is odder than anything else," said Ferrucci.

Watson could come to understand what an odditiy is over time, "by reading more material and playing more games," according to A Smarter Planet.

Jeopardy requires an enormously broad domain of knowledge with "infinite ways" to present the question, Ferrucci has said in the past. Even if Watson knew a topic well, it still needed to understand what the clue was asking for, he said. The categories of the first round were: Literary Characters APB, Beatles People, Olympic Oddities, Name the Decade, Final Frontiers, and Alternate Meanings. The literary characters category was heavy on puns and wordplay, and while Watson was supremely comfortable in the Beatles category, it got every single question in the decades category wrong.

Watson has an "Achilles heel," because its betting is restricted to its confidence level, according to "Final Jeopardy," a book about Watson by technology journalist Stephen Baker. Despite previous reports, Watson actually lost the practice match in January to Jennings in Final Jeopardy, because Jennings had bet aggressively on the Daily Doubles and Final Jeopardy, Baker said in an excerpt to final chapter of his book. The remainder of the chapter will be posted after the final game, and the hardcover of the book will be available on Thursday.