Author |
Topic: IBM AI plays Jeopardy (Read 10614 times) |
|
rbarreira
Forum Guru
Arimaa player #1621
Gender: 
Posts: 605
|
 |
Re: IBM AI plays Jeopardy
« Reply #30 on: Feb 17th, 2011, 4:09pm » |
 Quote  Modify
|
I think both the human contestants and Watson get the question text immediately, so all that's left to complain about in that regard is possibly that humans are slower at reading text.
But that would be a specious argument too, since Watson must also take some time to analyse the meaning of the question and might even evaluate it several times in a feedback loop with the search algorithms (just a guess). Also, if you give the text later to Watson it starts looking like giving a handicap to favor the humans, reminding us of the Kramnik vs Deep Fritz match...
I do somewhat agree with the "buzzer" concern though, that's probably because I think about Watson's feat as being language analysis rather than having fast mechanical devices (which is of course old technology and therefore shouldn't be the focus here).
|
| « Last Edit: Feb 17th, 2011, 4:21pm by rbarreira » |
 IP Logged |
|
|
|
Janzert
Forum Guru
Arimaa player #247
Gender:
Posts: 1016
|
|
Re: IBM AI plays Jeopardy
« Reply #31 on: Feb 17th, 2011, 4:51pm » |
Quote Modify
|
While I certainly agree Watson has an advantage in buzzer pressing response time, I'm not sure it's clear to everyone here that it was indeed pressing a mechanical switch the same as a human contestant does.
Also anyone/thing relying on speech recognition to get the clue from Trebek's reading of it is going to be at a huge disadvantage. Certainly almost all the players are reading it off the displays in the studio, not waiting to listen to the reading. You could argue Watson should have had to OCR it from cameras aimed at the displays too, but that doesn't seem very technically interesting.
Janzert
|
|
IP Logged |
|
|
|
Fritzlein
Forum Guru
Arimaa player #706
Gender:
Posts: 5928
|
|
Re: IBM AI plays Jeopardy
« Reply #32 on: Feb 17th, 2011, 5:57pm » |
Quote Modify
|
on Feb 17th, 2011, 4:09pm, rbarreira wrote:| I think both the human contestants and Watson get the question text immediately, so all that's left to complain about in that regard is possibly that humans are slower at reading text. |
|
You are neglecting a certain complication. In order to make Jeopardy suitable for television, the players are not allowed to hit their buzzers before Alex finishes reading the question out loud. Therefore, there will be questions for which Ken Jennings knows the answer a full second before Alex is done reading, but he is not allowed to buzz in. At some point, all the contestants get a signal "it is OK to buzz in now". Of course, if all three players know the answer and all three have already decided to buzz in, then Watson will win the reaction-time game to buzz in fastest after being told it is OK to buzz in. So what is being tested is not "how fast can you understand this question and come up with an answer", but rather, "how fast can you physically react to a signal".
Quote:| But that would be a specious argument too, since Watson must also take some time to analyse the meaning of the question and might even evaluate it several times in a feedback loop with the search algorithms (just a guess). Also, if you give the text later to Watson it starts looking like giving a handicap to favor the humans, reminding us of the Kramnik vs Deep Fritz match... |
|
In fact, the rules of Jeopardy handicap the humans by preventing them from buzzing in as soon as they know the answer. If you removed that handicap, gave all the contestants the question text simultaneously, and let them all buzz in whenever they were ready, then Watson would have come in third place by a mile, because the human champions can parse and understand the question faster.
Did you notice in the second day, in the category "Actor-Director", the humans got every question and Watson got none. Why? We could see Watson's thinking flash onto the screen, and Watson every time had the correct answer with a high confidence. The difference was that the questions were all very short, i.e. they were all just movie titles, so they didn't take long to read. This removed the handicap of having to wait for Alex to finish reading, so the humans beat Watson to the buzzer every time.
I am sure that some day the reverse would be true. Probably in a few years Watson could come back with the answer in under a second. For now, however, Watson would have lost a "fair" match. It is just a silly quirk of Jeopardy for television that all the contestants are artificially slowed down.
|
| « Last Edit: Feb 17th, 2011, 6:04pm by Fritzlein » |
IP Logged |
|
|
|
mistre
Forum Guru
Gender:
Posts: 553
|
|
Re: IBM AI plays Jeopardy
« Reply #33 on: Feb 17th, 2011, 10:35pm » |
Quote Modify
|
Thanks for clearing that up Fritzlein. It makes more sense to me now. Seems to me that the biggest variable to how well the contestants would do was question length then. Which makes for a very flawed match.
The only way to make it a fair match (without completely changing the rules of Jeopardy) would be to time the reaction time of Watson and tailor each and every question to take the same exact amount of time for Alex to read - so as to synchronize the time that the contestants would buzz in if they knew the answer.
|
|
IP Logged |
|
|
|
omar
Forum Guru
Arimaa player #2
Gender:
Posts: 1003
|
|
Re: IBM AI plays Jeopardy
« Reply #34 on: Feb 18th, 2011, 8:31am » |
Quote Modify
|
on Feb 17th, 2011, 5:57pm, Fritzlein wrote:
Did you notice in the second day, in the category "Actor-Director", the humans got every question and Watson got none. Why? We could see Watson's thinking flash onto the screen, and Watson every time had the correct answer with a high confidence. The difference was that the questions were all very short, i.e. they were all just movie titles, so they didn't take long to read. This removed the handicap of having to wait for Alex to finish reading, so the humans beat Watson to the buzzer every time.
|
|
Good observation Karl. I was thinking Watson was just not good at that topic, but your explanation makes more sense. While the rest of the world is happily accepting the computers victory, you've recognized the flaws in the Jeopardy game rules 
What could be done to make Jeopardy more fair in matches between humans and computers?
|
|
IP Logged |
|
|
|
megajester
Forum Guru
Istanbul, Turkey
Gender:
Posts: 710
|
|
Re: IBM AI plays Jeopardy
« Reply #35 on: Feb 18th, 2011, 8:59am » |
Quote Modify
|
on Feb 18th, 2011, 8:31am, omar wrote:
What could be done to make Jeopardy more fair in matches between humans and computers?
|
|
Put all contestants in separate rooms, ask the question, give humans and computers x seconds to formulate an answer and make them give it within 1 second of the buzzer sounding. Provide the computer with the question (or bits of it) in text form only once the humans have heard it (or heard enough to understand it).
EDIT: I only know Jeopardy from Groundhog Day so I hope I'm not making a complete idiot of myself. Still, if the whole point of the exercise is to see who can give the right answer fastest, then you have to make sure everybody understands the question at the same time and can buzz at the same speed when they have the answer. Giving the question to the computer only once the players have heard it sounds like a possibility for the first consideration. For the second, you could delay computer buzzing in line with human response times, eg. the amount of time it takes, say, a driver to push the break pedal after a hazard appears.
|
| « Last Edit: Feb 18th, 2011, 9:15am by megajester » |
IP Logged |
|
|
|
Sconibulus
Forum Guru
Arimaa player #4633
Gender:
Posts: 116
|
|
Re: IBM AI plays Jeopardy
« Reply #36 on: Feb 18th, 2011, 9:33am » |
Quote Modify
|
I'd think the easiest solution would be to permit early buzzing, and possibly take away the player's ability to read the whole question in advance. That's the way it was done in the vaguely jeapordish team trivia competitions in high school. In that case, you could buzz in whenever you wanted and try to answer, but you only lost points on an incorrect answer if you buzzed before the question had been entirely read... I suppose that is changing the game rather drastically though, and it would probably result in less game-time per episode aired unless the board size was increased.
|
|
IP Logged |
|
|
|
Fritzlein
Forum Guru
Arimaa player #706
Gender:
Posts: 5928
|
|
Re: IBM AI plays Jeopardy
« Reply #37 on: Feb 18th, 2011, 10:13am » |
Quote Modify
|
on Feb 18th, 2011, 8:31am, omar wrote:| What could be done to make Jeopardy more fair in matches between humans and computers? |
|
Well, "fair" is a really hard concept once you understand what each side can do well. For example, the question-writers would probably notice before to long that certain kinds of questions are more difficult for the computer to get correct. If they increased the percentage of such questions, the audience probably wouldn't notice, but it wouldn't be fair, would it? Also measuring buzz-in-reaction-time isn't "unfair", because it is the same for all players. It's just something machines happen to be good at.
I think I am less interested in a fair match than I am interested in an interesting match. The buzz-in-reaction-time issue doesn't bug me by virtue of being unfair, it bugs me by virtue of being obvious who will win. Yes, we know it takes a human 1/8th of a second to buzz in and a computer 1/100th of a second. Boring. Change the rules to make it unboring.
That said, on some level I respect the organizers for changing Jeopardy as little as possible to let Watson play. Whether the playing field of Jeopardy is naturally tipped to humans or to computers, at least they didn't change how much it is naturally tipped.
|
|
IP Logged |
|
|
|
megajester
Forum Guru
Istanbul, Turkey
Gender:
Posts: 710
|
|
Re: IBM AI plays Jeopardy
« Reply #38 on: Feb 18th, 2011, 10:20am » |
Quote Modify
|
Now why does this all remind me of the Kasparov v. Deep Blue match?...
New theory:
IBM hasn't taken on Arimaa yet because they're only interested in stuff they're guaranteed to win.
As indeed any hard-nosed businessman should be.
All IBM want to do is sell their top-notch hardware. The whole point of the Arimaa Challenge is to win using something other than top-notch hardware. Therefore if IBM ever take up the Arimaa Challenge I'll eat my hat.
I think more attention could be drawn to Arimaa (and from more worthy quarters than IBM) if it could be somehow demonstrated that computers by rights should be able to win the Arimaa Challenge. Which you can't do... without winning the Arimaa Challenge. Or can you?
|
| « Last Edit: Feb 18th, 2011, 10:30am by megajester » |
IP Logged |
|
|
|
Fritzlein
Forum Guru
Arimaa player #706
Gender:
Posts: 5928
|
|
Re: IBM AI plays Jeopardy
« Reply #39 on: Feb 18th, 2011, 10:27am » |
Quote Modify
|
on Feb 18th, 2011, 10:20am, megajester wrote:| IBM hasn't taken on Arimaa yet because they're only interested in stuff they're guaranteed to win. |
|
Heh, if that were so, then IBM wouldn't have taken on either chess or Jeopardy. Both were very risky and uncertain. What IBM really wants is good (free) publicity. But good publicity comes from doing something that millions of people think can't be done. It is hard to find something that millions of people think can't be done and you are guaranteed to win.
The trouble with Arimaa is that there aren't even a million people who know what it is, never mind a million people who think it can't be ruled by a computer. IBM will ignore us, but not primarily because Arimaa is an uninteresting challenge. So first we have to get millions of people hooked on playing Arimaa.
|
|
IP Logged |
|
|
|
megajester
Forum Guru
Istanbul, Turkey
Gender:
Posts: 710
|
|
Re: IBM AI plays Jeopardy
« Reply #40 on: Feb 18th, 2011, 10:50am » |
Quote Modify
|
on Feb 18th, 2011, 10:27am, Fritzlein wrote:
Heh, if that were so, then IBM wouldn't have taken on either chess or Jeopardy. |
|
OK you're right that they weren't guaranteed to win and you know the ins and outs better than I do. But I'm sure you'll grant me that no board of directors would have approved either without reasonable assurances from their experts that they were do-able.
Also, IBM sells hardware. By definition, the Arimaa Challenge removes all potential for showcasing hardware.
As you say, to attract any big players to the Challenge, we need to get millions of people playing Arimaa. And I'm hopeful that is possible. But if it does happen, I suspect they will all be watching because of their love of Arimaa, not their interest in AI. As I think you allude to in your book, and as Tiago Luchini over at BGG expresses in his own way, to promote Arimaa as just "chess tweaked so humans can still beat computers" sells the game short. It's a rich gaming experience that I know we all share, and I personally still have trouble putting into words. And I think it's that unique experience that will guarantee not only the prestige of the Arimaa Challenge, but the future of Arimaa.
|
| « Last Edit: Feb 18th, 2011, 11:01am by megajester » |
IP Logged |
|
|
|
The_Jeh
Forum Guru
Arimaa player #634
Gender:
Posts: 460
|
|
Re: IBM AI plays Jeopardy
« Reply #41 on: Feb 18th, 2011, 11:46am » |
Quote Modify
|
on Feb 18th, 2011, 10:27am, Fritzlein wrote:
Heh, if that were so, then IBM wouldn't have taken on either chess or Jeopardy. Both were very risky and uncertain. What IBM really wants is good (free) publicity. But good publicity comes from doing something that millions of people think can't be done. It is hard to find something that millions of people think can't be done and you are guaranteed to win.
The trouble with Arimaa is that there aren't even a million people who know what it is, never mind a million people who think it can't be ruled by a computer. IBM will ignore us, but not primarily because Arimaa is an uninteresting challenge. So first we have to get millions of people hooked on playing Arimaa. |
|
Of course IBM wants publicity, but I think the nature of the Jeopardy challenge as opposed to the Arimaa challenge has as much to do with IBM's choice as the size of the audience they will get. I'm not saying that the AI advances needed to play Arimaa effectively will turn out to be less usefully applicable (and hence less marketable) than those needed to play Jeopardy. I am saying that investors probably will see it that way, especially since chess has already been done. IBM has not proven their innovative prowess alone this time. Rather, they have demonstrated something that is almost an actual marketable product.
|
| « Last Edit: Feb 18th, 2011, 11:57am by The_Jeh » |
IP Logged |
|
|
|
rbarreira
Forum Guru
Arimaa player #1621
Gender:
Posts: 605
|
|
Re: IBM AI plays Jeopardy
« Reply #42 on: Feb 24th, 2011, 3:26am » |
Quote Modify
|
Kasparov has made some comments on Watson. On the whole his comments don't seem unreasonable, but it seems he still hasn't given up the idea that humans are superior to computers at chess... (I highlighted that part)
Quote: * A convincing victory under strict parameters, and if we stay within those limits Watson can be seen as an incremental advance in how well machines understand human language. But if you put the questions from the show into Google, you also get good answers, even better ones if you simplify the questions. To me, this means Watson is doing good job of breaking language down into points of data it can mine very quickly, and that it does it slightly better than Google does against the entire Internet.
* Much like how computers play chess, reducing the algorithm into "crunchable" elements can simulate the way humans do things in the result even though the computer's method is entirely different. If the result—the chess move, the Jeopardy answer—is all that matters, it's a success. If how the result is achieved matters more, I'm not so sure. For example, Deep Blue had no real impact on chess or science despite the hype surrounding its sporting achievement in defeating me. If Watson's skills can be translated into something useful, something groundbreaking, that is the test. If all it can do is beat humans on a game show Watson is just a passing entertainment akin to the wind-up automata of the 18th century.
* My concern about its utility, and I read they would like it to answer medical questions, is that Watson's performance reminded me of chess computers. They play fantastically well in maybe 90% of positions, but there is a selection of positions they do not understand at all. Worse, by definition they do not understand what they do not understand and so cannot avoid them. A strong human Jeopardy! player, or a human doctor, may get the answer wrong, but he is unlikely to make a huge blunder or category error—at least not without being aware of his own doubts. We are also good at judging our own level of certainty. A computer can simulate this by an artificial confidence measurement, but I would not like to be the patient who discovers the medical equivalent of answering "Toronto" in the "US Cities" category, as Watson did.
* I would not like to downplay the Watson team's achievement, because clearly they did something most did not yet believe possible. And IBM can be lauded for these experiments. I would only like to wait and see if there is anything for Watson beyond Jeopardy!. These contests attract the popular imagination, but it is possible that by defining the goals so narrowly they are aiming too low and thereby limit the possibilities of their creations. |
|
|
| « Last Edit: Feb 24th, 2011, 3:27am by rbarreira » |
IP Logged |
|
|
|
ddyer
Forum Guru
Gender:
Posts: 66
|
|
Re: IBM AI plays Jeopardy
« Reply #43 on: Feb 24th, 2011, 1:39pm » |
Quote Modify
|
on Feb 18th, 2011, 8:31am, omar wrote:
Good observation Karl. I was thinking Watson was just not good at that topic, but your explanation makes more sense. While the rest of the world is happily accepting the computers victory, you've recognized the flaws in the Jeopardy game rules
What could be done to make Jeopardy more fair in matches between humans and computers?
|
|
I don't find this plausible. Assuming humans and Watson are both "ready" before the allowed time to buzz in, Watson has nanosecond awareness of time and should be able to beat the humans every time.
In any case, the amazing thing about Watson is that Jeopordy becomes mostly a question of reflexes. To be that good at simulating natural language comprehension is awesome.
|
| « Last Edit: Feb 24th, 2011, 1:40pm by ddyer » |
IP Logged |
visit my game site: http://www.boardspace.net/
free online abstract strategy games
|
|
|
The_Jeh
Forum Guru
Arimaa player #634
Gender:
Posts: 460
|
|
Re: IBM AI plays Jeopardy
« Reply #44 on: Feb 24th, 2011, 2:06pm » |
Quote Modify
|
on Feb 24th, 2011, 1:39pm, ddyer wrote:
To be that good at simulating natural language comprehension is awesome.
|
|
"Simulating" is a good word. Watson is impressive with natural language input, but I don't know if I'd choose the verb "understand." If it can be made to "understand" one natural language, then it should be able to be made to "understand" two natural languages. In that case, IBM wouldn't be far away from translating between the two natural languages fluently.
|
| « Last Edit: Feb 24th, 2011, 2:07pm by The_Jeh » |
IP Logged |
|
|
|
|
Home
Help
Search
Members
Login
Register
Reply
Notify of replies
Send Topic
Print