Follow Slashdot stories on Twitter


Forgot your password?
IBM Entertainment Games Science

How IBM Plans To Win Jeopardy! 154

wjousts writes "Technology Review is reporting on IBM's plans to take on Trebek at his own game. The 'Watson' computer system uses natural-language processing techniques to break down questions into their structural components and then search its database for relevant answers. A televised matchup with Trebek is planned for next year. 'David Ferrucci, the IBM computer scientist leading the effort, explains that the system breaks a question into pieces, searches its own databases for "related knowledge," and then finally makes connections to assemble a result. Watson is not designed to search the Web, and IBM's end goal is a system that it can sell to its corporate customers who need to make large quantities of information more accessible.'"
This discussion has been archived. No new comments can be posted.

How IBM Plans To Win Jeopardy!

Comments Filter:
  • by eldavojohn ( 898314 ) * <> on Wednesday May 27, 2009 @11:08AM (#28109899) Journal
    I wonder how they plan to do with categories that have implications for all the answers. I've seen categories where words must be so many letters in length or perhaps start with certain things and Alex will interject while reading the category such as "'Cats'--and that means all the words in this category start with 'Cat'." Now, with that in mind, a clue could come in as "They are the popular makers of earth moving equipment." Might prompt Watson to find the most popular makers of earth moving equipment--Who is John Deere? The category of 'Cats' would do nothing for Watson without the aid of Alex's interjection ... thus failing at finding "Who is Caterpillar?" (bonus points if you also thought of "Who is Bobcat?" but that answer doesn't start with Cat).

    As a fairly avid though novice crossword puzzler, my mind explodes with questions. Could Watson discern a four letter word for "Pleasant French city" (Nice)? Or what about a four letter word for "Beefy Laker" (Kobe)?

    Lastly, will Watson have something inane and boring to talk about during the break?

    Alex Trebek: Now, Watson, it says here that you are named after Thomas J. Watson who forbade his employees to drink and even frowned upon it while off the job?
    Watson: That is correct. It is against IBM regulation 4-245 Section 8 to consume alcohol on the premises of any facility.
    Alex Trebek: Fascinating, I'm sure you've never broken that strict regulation, ha ha.
    Watson: Good sir, I am a computer, drinking is not within my capacity.
    Alex Trebek: Um, right. So could you tell us something interesting about yourself?
    Watson: *pauses to search records* During the fabrication of my circuitry, several engineers went months without sleep. Leading one to go insane and killed his wife and kid before taking his own life in a double homicide/suicide case.
    Alex Trebek: How unfortunate. Well, I wish you the best of luck today in Jeopardy.
    Watson: Thank you, my snide game show master.
    • Jesus (Score:2, Funny)

      What was an extra-terrestrial?

      • Jesus (Score:5, Funny)

        by eldavojohn ( 898314 ) * <> on Wednesday May 27, 2009 @11:17AM (#28110043) Journal

        What was an extra-terrestrial?

        How tastelessly incorrect. Extra-terrestrials don't come back to life. Watson would cross reference The Bible with many recent movies and come up with the correct question we were looking for: "What was a zombie?"

      • "My kingdom is not of this world" indeed...

      • by treeves ( 963993 )

        I know you were going for funny, but in reality Jeopardy never uses one(or two)-word answers. There are too many possible questions for a one-word answer.

        A: Caesar

        Q: Who crossed the Rubicon and said 'the die is cast'?
        Q: Who said 'Et tu Brute?' when he was assassinated?
        Q: Which Roman emperor adopted his great nephew Octavian who later became Augustus Caesar?
        Q: What salad is made with Romaine lettuce, anchovies, garlic and lemon?
        etc. etc.

        Instead you use the long answer and get a short question:

        A: This

        • by mopslik ( 688435 )

          I know you were going for funny, but in reality Jeopardy never uses one(or two)-word answers.

          They often do, but they're typically framed within a narrow, specific category. For example, the category might me "National Drinks" or some such thing. Typical Q/As might be Japan (Sake), Russia (Vodka), and so forth. Jeapoardy! also has a few other categories (anagrams, for example) that frequently use one or two words.

          • by treeves ( 963993 )
            Ah, you're right. I forgot about those cases. But they are exceptions, not the norm. I haven't watched Jeopardy! in a long time. I always thought I would have done well on Jeopardy! but not as well as that guy from Utah.
    • Re: (Score:3, Interesting)

      by wjousts ( 1529427 )

      Presumably they will either have to take into account the clues that come from the category itself (as in your example) or rig the system by avoiding "trick" categories. It's not an easy problem and it'll be very interesting to see what IBM come up with.

      An example from last night, they had a category "Knockouts" in both the first and second round. In the first round, all the answers were hot women (i.e. knockouts!), in the second round all the answers were about boxing. How will Watson deal with this? I don

      • by eldavojohn ( 898314 ) * <> on Wednesday May 27, 2009 @11:24AM (#28110149) Journal

        Presumably they will either have to take into account the clues that come from the category itself (as in your example) or rig the system by avoiding "trick" categories. It's not an easy problem and it'll be very interesting to see what IBM come up with.

        An example from last night, they had a category "Knockouts" in both the first and second round. In the first round, all the answers were hot women (i.e. knockouts!), in the second round all the answers were about boxing. How will Watson deal with this? I don't know.

        Yes, there are categories which require the contestant to have an active imagination and it's these categories I wish the article had addressed instead of a vanilla one. And I believe it's these categories that makes Jeopardy fresh and new after decades.

        In retrospect, I should have broke out the conversation into a different post so that this wasn't modded +5 Funny. I'm seriously interested in how IBM plans to address things that require the natural speech recognition of Alex Trebek. Does it take into account other answers in the same category to "catch on" like some contestants obviously do?

        Then there's the folks running Jeopardy who could pick some categories that would wreck Watson and give the humans the creative advantage. I hope they exploit this creative ability humans have and write an entire category in ... oh, say Pig Latin!

        In reality, they stand to have much more to gain if the machine comes close to winning ... as they could make this into an annual competition drawing fans and viewers much like the quest to beat the world chess grand masters.

        • by Ethanol-fueled ( 1125189 ) * on Wednesday May 27, 2009 @11:33AM (#28110281) Homepage Journal
          It's possible that the questions for that particular show will be specifically chosen to be more explicit and less ambiguous (avoiding the show's characteristic punny wordplay) to put the machine on a more level playing field, keeping its score closer to those of the contestants', which will make the episode more exciting to watch.
          • by eldavojohn ( 898314 ) * <> on Wednesday May 27, 2009 @11:36AM (#28110333) Journal

            It's possible that the questions for that particular show will be specifically chosen to be more explicit and less ambiguous ...

            Yes, clues like "It's the cube root of 474552" would level the playing field.

            Isn't the purpose of this to let Jeopardy be Jeopardy? And see if a computer can compete at what the show is?

            • by gnick ( 1211984 ) on Wednesday May 27, 2009 @11:50AM (#28110543) Homepage

              Without speculating on the specifics of tweaking the AI, my guess is that IBM has tried to think through these things. Having put together a few AI bots myself (purely recreationally - you know, just for kicks), I know that I let them play in the real world for quite a while to work out the kinks before unveiling them to nerdy friends and family to show them off and demonstrate just how much time and sleep I'd wasted. My poker-bot played thousands of games in free online rooms before I told anyone that I was even working on him.

              IBM has probably been feeding Watson DVR'd episodes for a while now so that they could identify (if not fix) the kind of gotchas that you're thinking about.

              • I'm on a mailing list with some participants in the Loebner Prize Contest, a version of the Turing Test. ("Robitron," if you want to see the list.) We were talking about this Jeopardy AI project recently.

                The most-discussed type of AI on that list is a "chatterbot" like ALICE or the classic ELIZA -- one that basically looks for key words and finds pre-written responses to them. That approach could probably tackle a variety of Jeopardy-type questions if someone took the trouble to feed the AI a bunch of sui
            • Re: (Score:1, Funny)

              by Anonymous Coward
              78 oh shit what is 78
            • by sexconker ( 1179573 ) on Wednesday May 27, 2009 @12:34PM (#28111235)

              Assume it's a perfect cube.
              x^3 is 6 digits, so we're looking at numbers from about 50 to 100.

              x^3 = 4XX
              6^3 = 216
              7^3 = 343
              8^3 = 512

              70 < x < 80

              x^3 ends in an 2, so the cube root must end in an 8.

              Seriously though, square roots are easy peasy.
              Cube roots let you use the awesome property that:

              0 - 0
              1 - 1
              2 - 8
              3 - 7
              4 - 4
              5 - 5
              6 - 6
              7 - 3
              8 - 2

              So you can always figure out the last digit of the cube root of a number VERY easily (no, you don't need to memorize that list).

              Then you use the size of the number to get a range, and then estimate. If you're feeling ballsy, you can go for it. Spend the first few seconds (before people buzz in) and get your range down. Then buzz in and spend a couple seconds estimating, then answer (just say "what is..." right when you buzz in). If someone else buzzes in first, more time for you to think.

              4th powers are just doing the square root twice.

              The list for 5th power roots is neat, too.

              0 - 0
              1 - 1
              2 - 2
              3 - 3
              4 - 4
              5 - 5
              6 - 6
              7 - 7
              8 - 8
              9 - 9
              0 - 0

        • Re: (Score:3, Interesting)

          by sexconker ( 1179573 )

          Speech recognition?
          The machine will be receiving a text file of the question.

          Hell, I bet the thing is always the first to the buzzer too.

          • Re: (Score:3, Interesting)

            by weszz ( 710261 )

            It may not be... would it hit the buzzer and hope it had time to compute the answer like many people on those shows, or would it wait until it had time to compute, and then ring in only if it has the answer?

            • Seeing as it can read the question instantly, instead of having to wait for Trebeck to read it out (or read it as text at human speed), I think it's got a serious advantage.

              I don't think the computational I/O speed is really going to be a limiting factor in this. The crux will be the dataset it has and whatever kind of strategy (being more or less gutsy with the buzzer, bidding more or less on the daily doubles / final round, etc. based on confidence of being right) it uses.

              It's not like playing chess agai

      • Well shit, I thought the 1st round was going to have knockouts (boxing) that occurred during the first round, and in the second round, the category would be about knockouts occurring in the second round.

      • Well knockout is a synonym for "hot babe". Shouldn't be too hard (of course I'm sure there will be something else that will be very hard for it)

    • Alex will interject while reading the category such as "'Cats'--and that means all the words in this category start with 'Cat'."

      Then the bot would read the closed caption that the category is "CATS MEANING ALL RESPONSES HAVE A WORD THAT STARTS WITH CAT" and include that in its reasoning. Then the clue "They are the popular makers of earth moving equipment" becomes something like "They are the popular makers of earth moving equipment, starting with 'CAT'".

      • Hm... one of the categories could be "Cats", you say...

        That must be why IBM didn't want the machine to search the Internet!

        - RG>

    • Re: (Score:2, Interesting)

      Since you bring up crosswords as an example of this sort of issue, let me point you to []

      Its an automated crossworld puzzle solver. How it works (and my advisor led the project, though I don't work on anything remotely similar) is that it has a large number of solver modules that are each good at a certain kind of clue. One might be really good at looking up famous people based on keywords. Another might be good at... I dunno some other type of crossword clue.

      Then each

    • by PMuse ( 320639 )

      I wonder how they plan to do with . . . categories where words must be so many letters in length or perhaps start with certain things . . .

      If they don't play one or more of these categories, it won't be full-up Jeopardy!(tm).

    • Seeing as you've thought of this by idly considering the problem, I would be extremely surprised if people who have spent months of research on the problem didn't already cover this one.

  • by Anonymous Coward

    I fed all the Jeopardy questions into Wolfram|Alpha and it got every single one right.

  • Only if... (Score:5, Funny)

    by weszz ( 710261 ) on Wednesday May 27, 2009 @11:11AM (#28109953)

    It can answer in Sean Connery's voice and make your mother jokes at him.

    Otherwise I'll probably pass and look up old SNL skits on youtube instead.

    • by SterlingSylver ( 1122973 ) on Wednesday May 27, 2009 @11:32AM (#28110271)

      So I think IBM's plans here are to
      Use a high-tech set of
      Computers to create a
      Knowledge processor that can be monetized.

      I think
      That wanting

      To use such a
      Rediculously advanced
      Engineering marvel to make Sean Connery jokes would
      Be a waste of
      Everone's time, energy, and

      • by weszz ( 710261 )

        think of it this way, if it can make jokes as well as being that good, I believe it would make more money and add some unexpected comic relief to things making it that much more valuable.

        you don't need to put your A people on it, just some guys that know what they are doing may do it on their time as well...

    • Re: (Score:3, Funny)

      by Ogive17 ( 691899 )
      IBM> You'll rue the day you crossed me Trebek!
  • by Anonymous Coward

    They plan to answer "Kebert Xela" and send that bastard back to the dimension where he belongs.

  • by Anonymous Coward on Wednesday May 27, 2009 @11:12AM (#28109967)

    I wonder how well it'll do at Anal bum cover.

  • by click2005 ( 921437 ) on Wednesday May 27, 2009 @11:12AM (#28109975)

    sell to its corporate customers who need to make large quantities of information more accessible.'"

    They want to replace the call centres in India with call computers.

    "Hello you're speaking to Susan Blue Gene how can I help you?"

  • Wordplay (Score:5, Insightful)

    by ooutland ( 146624 ) on Wednesday May 27, 2009 @11:19AM (#28110085) Homepage

    A lot of Jeopardy questions are wordplay-dependent, something AI doesn't have the hang of yet (unless IBM has been toiling in secret on something truly amazing). Categories like "Rhyme Time" and questions like "Qhat does a Pharoah need when he has a cold?" (Answer: an Egyptian Prescription) are beyond the ken of a data search.

    Many Jeopardy "answers" have the key to the answer within the question, though in some cases it may be enough to throw the program off. IE in a category like "Musicals" an answer like "Unlike his other hits, this musical wasn't 'the cat's meow' on Broadway." Raw data crunching will pair musicals, Broadway and "cats" but won't know where to go with "unlike." Only an aficionado will know that Andrew Lloyd Weber's "Starlight Express" tanked on Broadway.

    So the writers, given any knowledge of the limitations of AI, can set a challenge which will be nearly impossible for current AI to meet. John Henry will live another day.

    • Right. That's why it's interesting. It needs to interpret the questions, construct a query, process and rank the results, as well as store and index all the information it needs for the game (no live connection to the internet).

      If the questions aren't from the same group as usual, it won't be worth much. I would hope they wouldn't be specifically designed to either help or stump the computer.

      • by PJ1216 ( 1063738 ) *
        It says its not designed to search the web. This doesn't imply that its database must be stored locally. It can have a live internet connection, but only be talking to the database. I'd imagine the database is quite large.
        • Re: (Score:3, Informative)

          Sorry, I used information from outside the original article without a citation. This is from the team's web info []:

          ... just like human competitors, Watson will not be connected to the Internet or have any other outside assistance.

  • Waste (Score:2, Interesting)

    by Tablizer ( 95088 )

    IBM is laying off American citizens, but hiring in Asia, and yet are spending all this money on gimmicks. This is the kind of thing that gives big companies bad names. Hopefully, as a consolation prize, the laid-off Americans can watch their former company go down in smoke on the game show, hoping it starts smoking and sparking like a cheesy Trek android meltdown.

    • Re: (Score:3, Interesting)

      by wjousts ( 1529427 )
      It's not a gimmick. It is very important research on AI and natural-language processing. Jeopardy! just happens to give them a very difficult problem to tackle. If they can develop a system that can handle Jeopardy!, it'll be a huge break through for other fields.
    • Re:Waste (Score:5, Informative)

      by glwtta ( 532858 ) on Wednesday May 27, 2009 @11:48AM (#28110519) Homepage
      They're spending money on research, gimmicks just help pay for it.
    • Video Daily Double (Score:5, Interesting)

      by Comboman ( 895500 ) on Wednesday May 27, 2009 @12:07PM (#28110841)

      hoping it starts smoking and sparking like a cheesy Trek android meltdown.

      Alex: "Here to present the Video Daily Double is Harry Mudd, who always lies."

      Harry: "I am lying."

    • Well, since a big portion of IBM is consulting (50%?), it probably makes sense for them to hire consultants who are citizens of the countries they will be working in... How many Americans are fluent in a second language? How many are willing to relocate for 5+ years to Asia? I'd guess that hiring local gives IBM a GOOD name in those markets.

      Plus, the people they are hiring are human beings too. Why is hiring an American a more noble thing than hiring an Indian or Chinese person? They need to feed their

  • This isn't the first expert system I've come across called Watson and probably won't be the last.

    But has anyone pointed out to these guys that Holmes was the smart one? Watson just tagged along with him like a faithful puppy and generally gave little help in solving the crimes.

    So come on guys, how about a Holmes or Sherlock v1.0?

  • by sootman ( 158191 ) on Wednesday May 27, 2009 @11:24AM (#28110155) Homepage Journal

    The summary clearly should have been titled "How does IBM plan to win Jeopardy?"

    • by doomy ( 7461 )

      They will clearly win Jeopardy by patenting all the words in popular culture and using something like DMCA against all and any opponents... thus enforcing their patent argot.

  • A computer that can play Jeopardy?


    • by dzfoo ( 772245 )

      No, but you should start worrying as soon as it learns how to play Tic-Tac-Toe with itself. From there on, it's just a quick step into Global Thermonuclear War!


  • "I'm sorry, Watson. Your answer must be in the form of a question."

  • I hope "how many roads a can a man walk down..." is not a question

  • IBM just MUST make it sound like Sean Connery! Watson: I Google'ed your mother last night Trebeck!
  • by PJ1216 ( 1063738 ) * on Wednesday May 27, 2009 @12:02PM (#28110741)
    What is the answer to life, the universe, and everything?
    • Re: (Score:3, Insightful)

      by Bigjeff5 ( 1143585 )

      Well, we already know that, it's 42.

      The real question, is what is the real question for which 42 is the answer? That one is the tough one.

      I suggest we build a planet, who's sole purpose is to calculate that question...

    • Answer: The number of minutes it would take for gravity-powered travel between antipodes, and the angle in degrees which causes a rainbow to appear.

  • The system is not designed to access the web?
    Horse shit.
    That huge fucking pile of data is getting in there from the web. It won't be accessing the web during the game, but it's still a fucking cache of random shit (mostly geography and world history) from the internet.

    • So? Your point? It's not accessing the web live, is it? No? So it's not getting it from the web.

      The web isn't some magical information generating device. That information did not originate on the web, somebody put it there either from their own mind or from an offline location. By your own reasoning accessing the web isn't even "getting it from the web", as the web is just a huge cache of information from people's homes, schools, and even just plain their own minds.

      Which is actually, in a sense, correct,

      • Uh, no.
        A few interns were told to trawl wikipedia to grab data and form it into statements (probably prolog style).

        I guaranfuckingtee it.

        The game IS being rigged for the machine - the answers will be given to it in text. It won't have to press a physical button on a buzzer. Etc.

        • Ok, first, you totally missed my point. My point was that by extrapolating your objections to having a database of information obtained via the web, you actually get the data indirectly from the source, in some form. I don't see your objections to the human players who have been crawling for data for their own databases for the last 40+ years. Some of them have been crawling the web for data for the last 20 years!

          It's the same damn thing, data is data, half of Jeopardy is knowing shit. That's an area co

          • No, you missed my point.
            And you said so, yourself:

            "So? Your point?"

            Don't respond to my post, then put your own point in when you don't even get my point, then claim I don't get yours. Seriously, wtf?

  • So will IBM then try to get a patent for "Winning Jeopardy", then all the contestants have to pay royalties if they win?
  • All they need to do is use their super computers to generate some digital footage of Alex Trebek engaged in beastiality (tappin' Rosie O'Donnell) and then tell him that if they win, the footage disappears forever.

    It'd be far cheaper than what they are planning to do...and they can always leak the footage to youtube after they walk off with the winnings...

  • By searching for all the answer on [] because we know all the information on that site is correct!

    (Yes.. that was a joke)

    • Hey, anybody in the entire world can edit a wikipedia page, so you know you have the absolute best information possible!

  • It's a shame that 'Watson' works by breaking down questions into their structural components and then searching its database for relevant answers. After all, on Jeopardy all the answers are given freely.
  • by CHK6 ( 583097 ) on Wednesday May 27, 2009 @12:19PM (#28110991)
    So the new profit model for IBM is to make computers that can win game shows? Maybe someone needs to tell IBM that more money can be won in poker tournaments than Jeopardy. I can hear the board meeting conference, profit margins across our divisions were flat year over year, however our new game show competition division were good enough to bank roll our expensive luxury stay here on the island.
    • A good poker AI might actually be more interesting than a Jeopardy AI, even if the game it played was online so as to eliminate the factor of reading body language. For Jeopardy questions that boil down to "What is the capital of X" or "In what year did X happen?", a winning AI could basically just be Google running on an internal database. In contrast, winning at poker would involve social reasoning about questions like "When this guy suddenly raises his bet, is he often bluffing, and how likely is it that
  • In explaining the chain of reasoning, they weirdly left out that the name 'Pagliacci' is explicit in the lyrics, and proposed that Watson would deduce 'tears' as a form of feelings!? (Maybe they don't want to include a database of song lyrics?)

    They claim they won't use Web data, but there's no way they can compile enough databases on their own to handle Jeopardy's general knowledge. Awards, lyrics, plots, characters... the list goes on and on and on.

    WolframAlpha is a recent disappointment that's spent y

  • Here's a way to build a simple Jeopardy player that would kick a human's ass and doesn't require 4 years of programming:

    - Type entire "answer" as given on the board directly into google without quotes.
    - Search the returned page for the most common word (ignoring 2 letter ones) in the titles of the pages.
    - If the most common word appears more than 3 times, print "What is X?" where X is the common word.
    - If no one term appears that often, don't ring in.

    Voila. Instant human-crushing Jeopardy player.
    • You would think so, but it's not so simple.

      Google results are far from useful without context information, especially with Jeopardy where the context is usually hidden by a pun or a play on words.

      You just don't notice it because most people are able to process this contextual information on the fly, but it is a huge challenge for a machine to do it. There are litterally thousands of little bits of information that we collect as the answer is being given, including the context of the category, i.e. whether

    • Of course, you're solved it, you're so smart. Wait, I mean incredibly arrogant.

    • Easier...

      Hide Ken Jennings inside.

  • IBM has a history of inadvertently making terrible PR for themselves with these man-vs-machine stunts. Everyone here should remember Kasparov vs. Deep Blue. Expect IBM to win Jeopardy, and expect there to be a hailstorm of "IBM cheats" controversy after the game. []

  • What I want to know is this:

    The machine will probably be able to come up with an answer (maybe not the right one) much faster than all of the human opponents. But, what confidence will it have in that answer, and will it realize that a wrong answer will cost it?

    Obviously if the machine just answers immediately (and no 'confidence' factor is involved) then it could provide wrong answers very quickly, and thus just lose money on every question as it "presses the button" to answer the question before the oppo

  • Let's see if it can Win Ben Steins Money.

They are called computers simply because computation is the only significant job that has so far been given to them.