IBM Computer Program To Take On 'Jeopardy!' 213
longacre writes "I.B.M. plans to announce Monday that it is in the final stages of completing a computer program to compete against human 'Jeopardy!' contestants. If the program beats the humans, the field of artificial intelligence will have made a leap forward. ... The team is aiming not at a true thinking machine but at a new class of software that can 'understand' human questions and respond to them correctly. Such a program would have enormous economic implications. ... The proposed contest is an effort by I.B.M. to prove that its researchers can make significant technical progress by picking "grand challenges" like its early chess foray. The new bid is based on three years of work by a team that has grown to 20 experts in fields like natural language processing, machine learning and information retrieval. ... Under the rules of the match that the company has negotiated with the 'Jeopardy!' producers, the computer will not have to emulate all human qualities. It will receive questions as electronic text. The human contestants will both see the text of each question and hear it spoken by the show's host, Alex Trebek. ... Mr. Friedman added that they were also thinking about whom the human contestants should be and were considering inviting Ken Jennings, the 'Jeopardy!' contestant who won 74 consecutive times and collected $2.52 million in 2004."
Great... more phone-bots (Score:3, Funny)
Why employ real people when you can annoy the hell out of everyone who calls in by subjecting them to yet another tier of phone-bots.
Re:Great... more phone-bots (Score:4, Funny)
Why employ real people when you can annoy the hell out of everyone who calls in by subjecting them to yet another tier of phone-bots.
Or reverse it:
1) Use on telemarketers that call you. ...
2) Record
3)
4) Profit
The code name (Score:5, Funny)
Sources say the code-name for IBM's project is "Connery".
Trebek : This nobleman is believed to have written many of Shakespeare's works.
"Connery" : [pause] So that's your game, is it, Trebek? I was a coveted performer among the brothel ladies while you were still pissing your knee-pants, boy.
Trebek : Can one of the IBM people fix the computer?
"Connery" : Your mother's a whore. But don't feel badly, Trebek. She's not a very good one. Ha ha, ha ha!
Re: (Score:2)
Re:The code name (Score:5, Funny)
TREBEK: I'm sorry, that's "Japan US Relations." That's just awful and you know it.
Re:The code name (Score:5, Funny)
Alex Trebek: That's An album cover, not anal bum cover.
IBM: I can read, Trebek. That says Anal bum cover. I've spent five years of my life trying to invent an anal bum cover, failing to do so is my greatest regret
Re: (Score:3, Informative)
Alex Trebek: Yeah, it was a trick question, Mr. Connery. Why don't you pick a category?
Sean Connery: I've got to ask you about the Penis Mightier.
Alex Trebek: What? No. No, no, that is The Pen is Mightier.
Sean Connery: Gussy it up however you want, Trebek. What matters is does it work? Will it really mighty my penis, man?
Alex Trebek: It's not a product, Mr. Connery.
Sean Connery: Because I've ordered devices like that before - wasted a pretty penny, I don't mind telling you. And if The Penis Mightier works,
Re: (Score:2)
Re: (Score:2)
I was wondering what happens if the computer gets cracked or otherwise modified.
But then again, getting an advertisement for "low, low prices on hot teen pics" on jeopardy would be something of historical note.
I'll take The Rapists for $300 (Score:4, Funny)
That's "Therapists"
Leap Forward? (Score:3, Insightful)
In what way would this be a leap forward? Looking up trivial facts in a database and spitting them out is easy, and not particularly significant...
Re: (Score:2)
In what way would this be a leap forward?
Exactly what I was thinking. It sounds like they plan to connect voice recognition software to Google.
Re:Leap Forward? (Score:5, Informative)
According to TFA, the machine will get its questions as machine-readable text. The other human contestants will get it as text and audio. Also, the machine will not be connected to the internet.
Re:Leap Forward? (Score:4, Funny)
Sounds like they're guaranteed to win. The human contestants won't be given the questions at all :(
Re:Leap Forward? (Score:5, Insightful)
The step forward will be parsing the english language.
I hope it remembers to phrase its answers in the form of a question.
Re:Leap Forward? (Score:4, Informative)
I hope it remembers to phrase its answers in the form of a question.
Trivial. Word 1 of the question is "WHO" if answer is a person else "WHAT". Word 2 is "ARE" if answer is plural else "IS". Parsing Mr. Trebek's "answer" in three seconds, as you pointed out, is the hard part.
Re: (Score:2)
To get to the other side.
Why did the chicken cross the road?
When?
Where?
How?
I'm sure there have been Jeopardy! "questions" that needed to be started with each of the interrogatives.
Re: (Score:2)
Actually if the machine were able to do that via Google in a reliable way that would be something pretty substantial. I don't think that AI is good enough at that for Jeopardy at this time.
Re: (Score:2)
Exactly what I was thinking. Googling for the answer would provide the question (heck, `I'm feeling lucky' would probably get you the correct response in the title of the returned page).
Re:Leap Forward? (Score:4, Funny)
Exactly what I was thinking.
Jinx! You owe me a Coke.
Re: (Score:2)
Re: (Score:2)
We'll need to see the logs that include seconds in order to see if a coke is warranted.
Technically, my post was before Prof.Phreak since the commenting system posts comments chronologically. However, we don't know if they were a millisecond apart or 59 seconds apart.
Re: (Score:2)
*You* might not, but us 5-digiters do. Mwahahahahah.
Comment removed (Score:5, Insightful)
Re: (Score:3, Insightful)
Re: (Score:3, Funny)
Re:Leap Forward? (Score:5, Informative)
Parsing the questions in natural language, which is the goal here, is however very much *not* trivial. Doubly so since the clues and questions in a Jeopardy! game are usually at least somewhat obfuscated, contain puns, double entendres, etc...
Re:Leap Forward? (Score:5, Insightful)
Doubly so since the clues and questions in a Jeopardy! game are usually at least somewhat obfuscated, contain puns, double entendres, etc...
This is exactly why this sounds so implausible to me. You often have to take the category name and weave it in with the question (or rather, answer). A lot depends not on the knowledge, but on the phrasing of the "queries". Give me one example of translation software which can translate entire paragraphs well.
It makes me wonder how much "stress testing" they've done, by taking old Jeopardy questions and seeing if the output would be considered "correct" by a human arbiter.
Re: (Score:2, Insightful)
That's the sort of thing that makes me believe that this team may be able to succeed.
When Deep Blue went up against Kasperov, who could it practice against? Nobody.
There are tens of thousands of Jeopardy! questions to go through before they start making up their own.
Re: (Score:2)
It could have played against pretty much any chess player (apart, obviously, from kasperov). And it could have done it and determined who won for itself.
You need a *human* to determine if the answer is correct in this case.
Re: (Score:3, Insightful)
You don't need a human to determine who won a chess match. Winning is absolute.
You need a human to judge if the answer to a question, its phrasing, and its context were correct.
Re: (Score:3, Insightful)
When Deep Blue went up against Kasperov, who could it practice against? Nobody.
That this got modded Insightful is the best argument yet for adding tags to /.
Re:Leap Forward? (Score:5, Informative)
That's the sort of thing that makes me believe that this team may be able to succeed.
When Deep Blue went up against Kasperov, who could it practice against? Nobody.
There are tens of thousands of Jeopardy! questions to go through before they start making up their own.
Well it did practice against other grandmasters, and it analyzed every game Kasparov had every played, where Kasparov went into the match blind.
Re: (Score:3, Funny)
Sure! From my handy-dandy English-to-Tech Manual-to-English translator:
According to precise how from unlikely sounding me hereto. Must needs question category name sewing needle rapprochement. Muc
Re: (Score:3, Insightful)
Parsing the questions in natural language,
Natural language?
Outside of a Jeopardy! gameshow, I have never heard anybody use the this type of phrasing.
"This is a reason for you not handing in your homework Johnny"
"Why is because my dog ate it, sir"
Yeah, sounds very natural :-)
Re: (Score:2)
Re:Leap Forward? (Score:4, Insightful)
The leap forward is not in being able to look up facts in a database, it is in being able to interpret written questions properly.
There's a lot involved in interpreting natural language, and so far computers have been a far cry from being able to do it well. It says something that these algorithms are being tested against Jeopardy answers, since those are not completely natural language either -- they've been screened and edited to remove ambiguity.
Comment removed (Score:5, Insightful)
Re: (Score:2)
Rhyming categories, anagrams, the dreaded Before and After, slightly stupid answers, one letter off...
There's loads of categories they come up with that don't just rely on trivia knowledge but how to interpret the question.
Re: (Score:2)
hell, you can SQL query with Soundex nowadays.
Re: (Score:2)
Re: (Score:3, Interesting)
What msbmsb said. Also, this sounds similar to the problem already tackled (and aced) by Google Sets (or whatever they call it now). That's the feature where you give it some members of a set you have in mind (but you don't tell it what it's a set *of*) and it outputs more members from that set. For example, you give it "apple, orange, banana", and it gives you "grape, strawberry, lime". I'm guessing the way Google sets works is:
1) Run a search on all input phrases.
2) Find the most common statistically-
Re: (Score:2)
Both game playing and language processing are considered problems that full under the domain "AI."
Re:Leap Forward? (Score:5, Funny)
Well, at least a computer program will bother to RTFA.
Re:Leap Forward? (Score:5, Insightful)
Being able to beat a human at Jeapordy is a fairly substantial subset of the Turing test sorted.
Natural language processing is an absolute and total bitch - take it someone who has studied it. One of my AI professors once explained it to me such; the human brain tricks you into believing the hardest tasks it accomplishes are the easiest. Stuff like language, walking, and so on take up far much more of your neural hardware than what you would consider 'thinking' - but it all happens subconsciously.
No, it isn't Artificial Intelligence per se - there is no real 'understanging' or 'intelligence' behind it -but it is a very serious technical challenge. There is a lot more to it than simply dumping Jeopardy questions into a standard search engines.
Don't take my word for it. Load up your favourite editor or IDE and start coding a simple chat bot. The difficulties that IBM must have overcome are best discovered through experience.
Re: (Score:2)
Looking up trivial facts in a database and spitting them out is easy, and not particularly significant...
The significant part here is that the database queries will be in natural language instead of SQL and they will be kind of vague. Which is kind of a big deal, considering its one of those areas where pretty much all search engines these days fail at, i.e. you can't ask questions to Google and get an answer, instead you simply do full text search and get all pages matching, no matter if they have anything to do with your question or not.
Wierd (Score:5, Funny)
Anyone else hearing "I Lost on Jeopardy" in their heads at the moment?
Re:Weird Al (or is it A.I.?) (Score:5, Funny)
Anyone else hearing "I Lost on Jeopardy" in their heads at the moment?
Now I am. Thanks for that. Jerk.
Re: (Score:2, Funny)
I was there
To match my intellect
On national TV
Against an AI
Backed with a database
Both made by PhDs...
Hmm.. (Score:3, Funny)
Re: (Score:2)
What is eeeeppiiiiiikkk [[lost carrier]]?
Jeopardy really that challenging? (Score:2)
I know the exercise is not in the google-fu of a computer but in its ability to interpret Trebeck's questions as well as answer in the right form but Jeopardy hints and questions are very well-formed. That is, it doesn't contain much if any of the ambiguities of normal speech.
"This city was formed by the brothers Romulus and Remus"
Answer "What is Rome"
Seems a fairly easy speech pattern. A more interesting challenge would be Who Wants to be a Millionaire.
see the question here .. (Score:2)
Re: (Score:2)
I'm pretty sure it would get the elephant/moon question wrong too.
"Which is bigger, the elephant or the moon?"
Obviously, the moon is bigger, but to provide that answer requires the knowledge that "bigger" in this case meant actual size, not appearance.
Re: (Score:3, Insightful)
Actually, yes, it is. See, it's not just general knowledge, but, as about 9 billion other people in this thread pointed out, there are puns and other wordplay often involved.
What do you think the proper Jeopardy answer to this question is (in the category "Much Ado" for $100):
"It's the spirit that gets things done."
Answer: What is "can do"
The $500 version might be something like, "This recently hip-again party favorite was first created in New York."
Answer: What is "fondue"
Both of those are pretty easy exam
Re: (Score:2)
> A more interesting challenge would be Who Wants to be a Millionaire.
All expectations would be on IBM's advanced AI winning Who Wants to be a Millionaire. Yet despite those expectations we would be thrilled to see a young, plucky A.I. that grew up in dirt poverty in the video game arcades of Detroit manage to win through the cooincidence of the questions all relating to events in the young A.I.s life; Questions such as "In Defender, the protagonist rides on the back of (A) An Elephant (B) A Horse (C) An
Re: (Score:2)
I'd take Jennings (Score:4, Insightful)
The problem they might run into is the speed of pressing the button to respond. I would imagine the computer would be able to beat the human every time it knew the answer.
Re: (Score:2, Informative)
Re: (Score:3, Interesting)
The problem they might run into is the speed of pressing the button to respond. I would imagine the computer would be able to beat the human every time it knew the answer.
This is actually where I think the humans have an advantage. They can press the button just because they think that they WILL know the answer in the time allotted.
Watson may be designed to predict its own ability to answer. But to allow it to just press the button, then use the entire time limit to find it would not be fair...
Re: (Score:3, Insightful)
It would be perfectly fair. If it doesn't come up with the answer, it loses money. Same rule the human players are under.
Wolfram Alpha (Score:2)
And does this really fall under "supercomputing"? Couldn't this be done in a distributed fashion?
Jeopardy doesn't work that way (Score:5, Funny)
I feel like someone should tell them how Jeopardy works... That thing isn't going to have too many questions to respond to.
Except at the "meet the contestant" part, maybe, which by the way should be fascinating.
I have a prediction for the meet the contestant... (Score:3, Funny)
Except at the "meet the contestant" part, maybe, which by the way should be fascinating.
"So, computer, you're about two months old, and you grew up in IBM's labs, right?"
"Bite my shiny metal ass"
Is this what we really want? (Score:2)
Re:Is this what we really want? (Score:4, Funny)
Subjugated Human: What is my home?
Robot: That is the incorrect question. Please follow me to a "processing station".
Great!!! (Score:2)
Two words... (Score:2)
if this computer uses Google, (Score:2, Funny)
This is how it starts (Score:5, Funny)
Re: (Score:2)
He's BACK!
Re: (Score:3, Funny)
Re:This is how it starts (Score:5, Funny)
new objective: to win all human gameshows and use the prize money to buy off the entire planet instead.
I think Pinky and the Brain [wikipedia.org] already used that plot device.
Re: (Score:3, Informative)
And... then... maybe... (Score:2)
Stand by your sentences. End them with a single punctuation mark like a real man.
Re: (Score:2)
Re: (Score:2)
The ellipses mean, "there was some other text between these sentences, but we stripped that out, because it was not relevant to our point."
It does not indicate uncertainty on the part of the speaker. Or that the speaker was William Shatner.
Logs (Score:3, Interesting)
logs or it didn't happen! what they did to Kasparov was bullshit! Seriously if this magically gets better at 1/2 time, the least they can do is show the logs
Comment removed (Score:3, Informative)
Re: (Score:2)
Re: (Score:2, Funny)
Um... hello? Jeopardy is a trivia game show.
j/k
Comment removed (Score:5, Insightful)
Does it take more than eleventy billion... (Score:2)
...calculations to come up with "Oh, I'll bet you do, you Canadian ponch."?
--------
stupid subject character limit
Buzzing In (Score:3, Interesting)
Mod parent up (Score:2)
And me without my mod points!
Yea, that's one of the great challenges, and if you ever watch a high-caliber contest (like the recent Tournament of Champions), you'll notice that the buzzer timing often plays a MORE important role than the actual knowledge.
But this whole IBM thing is just theatrics anyway. The computer has impeccable timing and a limitless database of knowledge. All they are proving is that it can recognize and parse human speech. But they don't need Jeopardy! for that. They could demonst
I've already done this: man -vs- machine (Score:2)
I had a coworker who was very "special" - socially awkward, multiple degrees, very brilliant, and only slept a few hours a day. I called him Cliff Clavin [wikipedia.org] as he was a master of trivia. We had a game where we would pit his brain against Google + my typing skills. Throughout the day he would randomly stand up and announce some esoteric fact. I would then Google the fact, and try to present another fact equally esoteric. We would go about this until I either stumped him, or he outpaced me. It was very ent
But is it this witty? (Score:2)
Seriously? I thought they should have given him the cash for the answer. Sounds about right to me.
Already solved (Score:2)
Category with the answer in the question (Score:2)
There is a Jeopardy category where part of the answer is in the category name. Eg would could it "bee"? Where all the answers have the string "bee" in the answer. That kind of question would be easier for a machine than a person.
First test run (Score:2)
Host: The first man to walk on the Moon.
Computer: Neil Armstrong.
Computer: I mean, who is... Doh!
Can I have theatrics for $2000, Alex? (Score:3, Insightful)
So... why Jeopardy? IBM is trying to demonstrate software that can parse text for meaning. That's great. But there are plenty of other places/formats/etc. that you can demonstrate this technology. There are certainly far more useful applications of this sort of technology.
I'm guessing that they they are going after J! because...
1) The warm spotlight of a well-known TV show
2) There is still a lot of structure and form on J! that it's easier to achieve "success" than if they had the machine do something more free-form... e.g., read a novel and generate a plot summary or, heavens forbid, actually understand real human conversation
3) The computer could have other advantages, like impeccable buzzer timing (which is sometimes more important than actual knowledge, especially in the Tournament of Champions) and having memorized the material beforehand (the NYT indicates that it would have "read" study materials before the match), which also helps increase the likelihood of "success"
And to pile on the criticism of grandstanding, the machine will be fed electronic text. So no video camera to perform text recognition? No speech recognition (IBM afraid of the "wreck a nice beach" vs. "recognize speech" problem tripping up their theatrics?). And what use would this be? At least the AI text research done at Google is being put to good use, like improving their machine translation services. Aside from getting IBM's name plastered in the media, what exactly is this going to do?
game show is gonna be a real wopper (Score:3, Funny)
Mr. Trebek, would you like to play a game?
Possible Categories: (Score:2)
Possible Categories:
"Self-Awareness, I Haz it!" ...
"Die Humans, Die, Die, Die!"
"Computations of Pi"
"All your Base"
That's why IBM didn't buy Sun . . . (Score:4, Funny)
They had to make some tough investment decisions: Either buy Sun, or build a Jeopardy playing supercomputer, but not both.
I'm sure that the machine's performance in Final Jeopardy will awe us sufficiently, and IBM's management will be exonerated from walking away from Sun.
IBM: "Hey, Larry Ellison! Have fun in your toy sailboat! Call us when your database and hardware synergies can compete with us in pre-prime time light entertainment game shows!"
Rumor has it that HP is working on a massively parallel Intel supercomputer that can calculate the strategic advantage of bidding one dollar ($1) on "The Price is Right."
Head to head with IBM (Score:3, Funny)
We were about even going into final Jeopardy when he stubbornly refused to offer any question for the answer "Smartest ever computer in the movies". I got it right (HAL) and took the prize.
Total geek-gasm (Score:3, Interesting)
The way to deal with such problems, Dr. Ferrucci said, is to improve the programâ(TM)s ability to understand the way Jeopardy! clues are offered. The complexity of the challenge is underscored by the subtlety involved in capturing the exact meaning of a spoken sentence. For example, the sentence "I never said she stole my money" can have seven different meanings depending on which word is stressed. "We love those sentences," Dr. Nyberg said. "Those are the ones we talk about when weâ(TM)re sitting around having beers after work."
Seriously guys, I just had a geek-gasm. Anyone else?
State of the Art Voice Recognition (Score:5, Interesting)
We were just talking about this in another thread... A lot o the comments here have been that natural language software isn't that great.
This isn't at all true. Today, understanding verbal and written communication is done by state of the art computers and programs at a rate about equal to human listeners and readers. Where a computer doesn't particularly excel is in parsing that language, mostly because a computer doesn't have access to our culture in order to absorb context, but context can still be added.
Here's an example from Ray Kurzweil's book "Age of Spiritual Machines." He talks about a phrase famously given to a language parsing program that goes thus: "Times flies like an arrow." This phrase can be understood in various ways:
"* The common simile: time moves quickly just like an arrow does;
* measure the speed of flies like you would measure that of an arrow (thus interpreted as an imperative) - i.e. (You should) time flies as you would (time) an arrow;
* measure the speed of flies like an arrow would - i.e. Time flies in the same way that an arrow would (time them);
* measure the speed of flies that are like arrows - i.e. Time those flies that are like arrows;
* all of a type of flying insect, "time-flies," collectively enjoys a single arrow (compare Fruit flies like a banana);
* each of a type of flying insect, "time-flies," individually enjoys a different arrow (similar comparison applies);
* A concrete object, for example the magazine, Time, travels through the air in an arrow-like manner."
(from: http://en.wikipedia.org/wiki/Natural_language_processing [wikipedia.org])
With a few facts it becomes obvious which is correct and which isn't. Tell the computer that there's no such thing as a 'time fly' and that flies don't time things, and that flies aren't sophisticated enough to like things in an affectionate manner, etc., and the correct interpretation soon becomes clear.
So, if you think a computer can't understand both written and verbal communication and then parse it quickly enough to answer the questions I will have to strenuously disagree. These challenges are quickly being overcome on the bleeding edge of the art. But since this perception persists that the state of the art is somehow bad, because Joe down the street messed with some free-ware language software that worked poorly -- I think a lot of people are in for a surprise, and winning Jeopardy in this manner is really the perfect way to show it off. Can't wait to see the Youtube clips.
Re: (Score:2)
AI doth never prosper, what's the reason? If it prosper, none dare call it AI.
We've been seeing this since Turing and the Turing test. If it is an unsolved, apparently open ended problem, it must be an AI application. As soon as someone figures out how to do it, it wasn't really AI after all, just an obscure, but somehow also mainstream, technique to be worked out. They move the goal posts and say AI is now over there in that other unsolved problem.
Re:Not AI just Google with a filter (Score:4, Insightful)
No argument will show you how wrong you are - please just try to code something similar. Or even a chat bot. Things like language processing and image recognition seem easy to humans because most of your brains activity is hidden from your conscious mind. Try making a computer do these things, and you will discover that what IBM is nontrivial.
Neither was selling calculating equipment to the Nazis either, like you said.
Re: (Score:2)
I don't agree, having watched jeopardy in the past, I can definitely say that if you're just down to the "who" or "what" you're doing quite well on some of those categories, particularly towards the end of the show.
A computer is going to have a hard time getting to that point in the process. It's trivial to do, just know whether the subject is a person or not a person. Admittedly that would throw off IBM's 40s era algorithms when applied to Jewish people, but I doubt they're using anything other than the mo