IBM's Jeopardy-playing supercomputer is now capable of beating human Jeopardy contestants on a regular basis, but has a ways to go before it takes on the likes of 74-time champion Ken Jennings.
IBM announced plans to build a computer that can win on Jeopardy last April, and expects to stage a public tournament involving human players and the machine within the next year or so.
The question-answering system, nicknamed "Watson", is already doing trial runs against people who have actually appeared on the Alex Trebek-hosted Jeopardy. Watson's competition includes people who qualified for the show but lost, people who appeared and won once, and people who appeared and won twice.
Watson is "working its way up through the ranks," says David Ferrucci, leader of the project team. "We win some, we lose some. Overall, we're quite competitive but there's a ways to go to play the top of the top."
The games are played at IBM's "Watson Research Center" in Yorktown Heights, N.Y., with a real stage and professional host -- though not Alex Trebek. Questions are provided by Jeopardy, and the computer -- about the size of eight refrigerators -- is seen behind a glass window while the two human contestants are at podiums.
The computer has an advantage when it comes to ringing in on questions. Anyone who's seen Jeopardy knows that human contestants can struggle with timing when buzzing in after questions. Watson has great reaction speed.
But the Watson development team faces many challenges in creating a robotic Jeopardy champion. Without being connected to the Internet, the computer has to understand natural language, determine the answer to a question (or, in the case of Jeopardy, the question to an answer), and then calculate the odds that its answer is correct in order to decide whether it is worth buzzing in.
The fear of getting a question wrong and losing money prevents many a wrong answer from a human Jeopardy contestant. At the same time, humans often instinctively know they know the answer to a question, even if it doesn't pop into their heads right away. So a human Jeopardy player will often click the buzzer upon hearing the question, and then spend the next several seconds pulling the answer out of the memory bank. To compete on Jeopardy, a computer must determine whether it knows the correct response within seconds.
Watson also has to be programmed to play strategically. The computer may be reasonably sure it knows the answer to a question, but will take into account the score of the game and the dollar value of the question before deciding whether it is worth taking the risk. It also needs a strategy for choosing categories and a betting strategy for Daily Doubles and Final Jeopardy.
IBM has experience building artificial intelligence systems that can compete against humans. On Feb. 10, 1996, IBM's "Deep Blue" won a game against chess champion Garry Kasparov, but ultimately lost the match 4-2. On May 11, 1997, Deep Blue beat Kasparov in a full match.
Deep Blue's victory over Kasparov was an unprecedented achievement, but winning at Jeopardy with a computer is a very different task and likely a more difficult one, Ferrucci says. Chess is in many ways a well-defined mathematical problem. Winning at Jeopardy requires an understanding of natural language and the ability to make educated guesses and take calculated risks.
The very act of understanding a question is riddled with roadblocks. Words often have multiple meanings and can have different relationships with each other, and have to be interpreted in the proper context. Then the computer has to search within its own knowledge base to find an answer.
Watson isn't just one computer. IBM is experimenting with several high-performance clusters including a Blue Gene supercomputer and a Power7 cluster. Thousands of processing cores and terabytes of memory are needed to compete against human players. "This is a massively parallel computation, and IBM has lots of options for doing this kind of massive parallelism," Ferrucci says.
While the Jeopardy challenge is meant to be entertaining, Ferrucci says the underlying research could help solve much more important problems. The ability to mine large quantities of data and determine the accuracy of a conclusion could come in handy in business intelligence systems, as well as in medical diagnoses.
"As difficult as it will be for the Watson computing system to compete on the leading quiz show for brainy humans, the ultimate test will be to move beyond the Jeopardy Challenge," IBM says on its Watson Web site. "The goal is to have computers start to interact in natural human terms across a range of applications and processes, understanding the questions that humans ask and providing answers that humans can understand and justify."
IBM isn't ready to announce a date for a public Jeopardy challenge, and the company is not releasing statistics that would show exactly how well Watson has done against human opponents in test matches. IBM and Jeopardy are restricting many details of the development process as part of their agreement with each other, so when Ferrucci says Watson is "quite competitive" against former Jeopardy players it's not quite clear what he means.
A Jeopardy contest has three players, so if Watson can win more than one-third of the time it could be said that it is better than the average human player. But when asked if Watson has won at least a third of its games, IBM officials said they're not ready to announce that.
Ferrucci can say one thing without violating any nondisclosure agreements: "It's been fun to watch."
Follow Jon Brodkin on Twitter: www.twitter.com/jbrodkin
Read more about data center in Network World's Data Center section.
This story, "IBM's Jeopardy-playing machine can now beat human contestants" was originally published by Network World.