Arimaa Forum
        (http://arimaa.com/arimaa/forum/cgi/YaBB.cgi)
      

        Arimaa >> Off Topic Discussion >> IBM AI plays Jeopardy
        
(Message started by: Fritzlein on Jul 16th, 2010, 8:36am)
Title: IBM AI plays Jeopardy
Post by Fritzlein on Jul 16th, 2010, 8:36am
http://www.nytimes.com/2010/06/20/magazine/20Computer-t.html

Bad news folks.  Deep down I have long cherished a hope that Arimaa would eventually attract the attention of IBM.  Maybe not right away, maybe not until the player base grows and the Arimaa Challenge remains unconquered through 2020, but some day.  Particularly if the man/machine rating gap happened to be even bigger in 2020 than it is today, say 800 rating points instead of 500, perhaps IBM would think Arimaa was interesting enough to take on.

Alas, my hopes, faint and long-term though they were, have been crushed.  IBM has now used a supercomputer to create a competent Jeopardy player.  Apparently they aren't quite up to Ken Jennings' level yet, but they can beat most Jeopardy contestants already, and are planning a televised showdown with past champions.

Even if IBM loses now, they will focus all their "long-shot game-playing" attention on Jeopardy, because it is ten times cooler than playing mere board games.  Whether or not IBM could beat the human Arimaa champion in 2021 with a supercomputer, everyone will assume they could, because Deep Blue beat Kasparov.  Been there, done that.  Jeopardy, on the other hand, is new, exciting, and scary.  When AI is doing cool stuff like that, no AI money will ever chase Arimaa.  

:'(
Title: Re: IBM AI plays Jeopardy
Post by rbarreira on Jul 16th, 2010, 9:04am
That is really impressive when you see the kinds of problems Watson managed to solve.

PS: The people who thought Deep Blue was cheating at chess are going to have a field day with this.
Title: Re: IBM AI plays Jeopardy
Post by Arimabuff on Jul 16th, 2010, 10:21am
What's so tricky about Jeopardy? It's just a stupid question/answer game with the requisite that you have to formulate the answer as a question, a reverse trivial pursuit so to speak. Frankly I have never watched much of this because I find it a tad annoying but anyway, I don't see what the fuss is all about. Put everything  trivia like in a database  plus a program that will put the data in the form of a question (how complicated can it be?) and voilŕ!
Title: Re: IBM AI plays Jeopardy
Post by Fritzlein on Jul 16th, 2010, 11:02am
Answering in the form of a question is not the major issue.  The trick is understanding what information is being fished for.  Give the article a read.  It explains the gap between questions of a predefined form such as "What is the capital of Denmark?" and free-form questions involving implications like, "What was relatively important, compared to knowledge, to relativity's founder?"

The latter question can be answered by someone who knows that Einstein founded the theory of Relativity and also knows that Einstein said "Imagination is more important than knowledge," but is isn't something that can be stuffed in a database in advance.
Title: Re: IBM AI plays Jeopardy
Post by Hirocon on Jul 16th, 2010, 3:08pm
I'm still waiting for a team of humanoid robots that can beat a human team at soccer (aka football).
Title: Re: IBM AI plays Jeopardy
Post by The_Jeh on Jul 16th, 2010, 4:46pm

on 07/16/10 at 10:21:30, Arimabuff wrote:
It's just a stupid question/answer game with the requisite that you have to formulate the answer as a question, a reverse trivial pursuit so to speak. Frankly I have never watched much of this because I find it a tad annoying but anyway,


Take it back! It's the greatest quiz show of all time. I watch it religiously.   :)

Just as a bit of trivia - the original reason Jeopardy made contestants give answers in the form of questions was to poke fun at the quiz show scandals that came before it in the 1950's. It was found that producers had been rigging games by giving contestants the answers. So, why not make them give the questions instead? As it turns out, it really gave the game a great flow.

Anyway, I'm impressed that with so much knowledge to process a computer could still answer in 6 or 7 seconds.
Title: Re: IBM AI plays Jeopardy
Post by rbarreira on Jul 16th, 2010, 9:33pm

on 07/16/10 at 15:08:48, Hirocon wrote:
I'm still waiting for a team of humanoid robots that can beat a human team at soccer (aka football).


That doesn't need any AI, as long as you allow robots which move very fast :P
Title: Re: IBM AI plays Jeopardy
Post by Hirocon on Jul 16th, 2010, 9:56pm
Getting humanoid robots to move very fast requires a lot of AI; in fact in requires a lot more AI than in takes to play chess.  See:

RoboCup (http://www.robocup.org/)
Robocup Soccer 2010 Germany / Germany (Final) (http://www.youtube.com/watch?v=4wMSiKHPKX4)

That was state-of-the-art earlier this year, and those robots were not nearly as agile or fast as humans.
Title: Re: IBM AI plays Jeopardy
Post by rbarreira on Jul 16th, 2010, 10:00pm
I think that's more robotics/mechanics than AI. Sure, if your mechanical components are not very good you need more AI (as is the case with Robocup teams).
Title: Re: IBM AI plays Jeopardy
Post by Arimabuff on Jul 17th, 2010, 10:08am

on 07/16/10 at 16:46:04, The_Jeh wrote:
...

Anyway, I'm impressed that with so much knowledge to process a computer could still answer in 6 or 7 seconds.

Well the delay to access information in a database is pretty much a logarithmic (like) function of its volume that is why you can access information on the net in only a fraction of a second even though there are billions of pages to choose from. I believe that most of the 6 sec. You speak of is spent making complicated connections between fragments of data as in a compounded question/answer like the one Fritz, talked about. I’ve seen a program that could create a complex cross word puzzle, so much so in fact that it was beyond the reach of the best cross word puzzle makers in the world! And the program would do it in a few SECONDS!!! Makes you wonder, doesn’t it?
Title: Re: IBM AI plays Jeopardy
Post by Arimabuff on Jul 17th, 2010, 10:24am

on 07/16/10 at 11:02:07, Fritzlein wrote:
Answering in the form of a question is not the major issue.  The trick is understanding what information is being fished for.  Give the article a read.  It explains the gap between questions of a predefined form such as "What is the capital of Denmark?" and free-form questions involving implications like, "What was relatively important, compared to knowledge, to relativity's founder?"

The latter question can be answered by someone who knows that Einstein founded the theory of Relativity and also knows that Einstein said "Imagination is more important than knowledge," but is isn't something that can be stuffed in a database in advance.

I think it’s much less difficult than is seems. A program can analyze an expression like "relativity's founder" even though it's imprecise and incorrect, but a program can work with imprecise and incorrect data. Your word processor does it all the time when it corrects your spelling and your grammar, for example. So the program could probably get Einstein out of that expression and then do a keyword search on every quote that has ever been attributed to Einstein. I bet that there aren’t many that contain the words "important" and "knowledge" etc... That doesn't look too complicated to me.

I mean we already have programs that do amazing searches on approximate and sometimes even WRONG keywords and their synonyms in record time don't we?
Title: Re: IBM AI plays Jeopardy
Post by RonWeasley on Jul 18th, 2010, 3:57pm
What is a fast internet search engine?
Title: Re: IBM AI plays Jeopardy
Post by omar on Jul 21st, 2010, 6:06pm
Thanks for sharing that interesting story with us Karl.

I don't expect IBM to ever be interested in the Arimaa Challenge; especially since the challenge does not allow using supercomputers. I think part of IBM's reason for taking on chess and now Jeopardy is to showcase their supercomputers.

Although IBM may never try for the Arimaa challenge, the work they are doing here with Jeopardy could potentially be applied to Arimaa. In a way they are tackling the problem of not being able to generate or store a full position to move (question to answer in Jeopardy) lookup table by using statistical data mining on a large data set of information (like the Arimaa games archive). I think Toby has already started down this path with his bot GnoBot. Who knows maybe it will work :-) If it does, it will be a very different approach than the conventional tree search approach.
Title: Re: IBM AI plays Jeopardy
Post by The_Jeh on Feb 14th, 2011, 10:44am
I justed wanted to remind everybody that today, tomorrow, and Wednesday the duel among IBM's Watson, Ken Jennings, and Brad Rutter will be airing. I believe Watson already won a 15-question mini-game against the two, in which no one answered incorrectly but Watson buzzed in more quickly.
Title: Re: IBM AI plays Jeopardy
Post by ddyer on Feb 14th, 2011, 1:39pm
From an AI standpoint, the interesting thing about Watson is that it solves the "Jeopordy" problem in the same way that humans do.   It interprets an ambiguous question and weighs many factors to decide on both the meaning of the question and the probable correctness of many answers, and even judges its own certainty of it's result.

That is light years away from the methods used to play traditional board games like chess or Arimaa.
Title: Re: IBM AI plays Jeopardy
Post by Fritzlein on Feb 14th, 2011, 3:32pm

on 02/14/11 at 13:39:17, ddyer wrote:
That is light years away from the methods used to play traditional board games like chess or Arimaa.

Hmmm... I was just wondering whether there is some parallel on an abstract plane.  In both chess and Jeopardy the computer spends a lot of time on worthless answers.  There is the "answer generation" part of Watson (like search in computer chess) and the "answer assessment" part of Watson (like static position evaluation in chess).  Of course it seems to me that both generating answers and evaluating answers for Jeopardy questions would be harder problems than the analogous bits of chess engines, but somehow it seems to me that there is a structural similarity.

Chess programs have been disparaged as being dumb and fast, but for all we know, this is how our own brains work without our knowing it.  Maybe when someone asks, "Who was president before Kennedy?", our brains try out and reject a million different answers without our being aware that it is going on, because all that we eventually become conscious of is the most probable answer that passes plausibility filters and rises to the top.
Title: Re: IBM AI plays Jeopardy
Post by omar on Feb 14th, 2011, 6:52pm
I watched the show today. Watson was quite impressive. IBM should consider putting it online to allow people to ask it questions.


on 07/16/10 at 08:36:21, Fritzlein wrote:
Bad news folks.  Deep down I have long cherished a hope that Arimaa would eventually attract the attention of IBM.  Maybe not right away, maybe not until the player base grows and the Arimaa Challenge remains unconquered through 2020, but some day.  Particularly if the man/machine rating gap happened to be even bigger in 2020 than it is today, say 800 rating points instead of 500, perhaps IBM would think Arimaa was interesting enough to take on.


Maybe IBM already looked at Arimaa and decided it was too difficult :-)

https://www-927.ibm.com/ibm/cas/hspc/Profiles/People/profiles.shtml#Newborn
Title: Re: IBM AI plays Jeopardy
Post by aaaa on Feb 14th, 2011, 7:57pm
The first episode has been put on YouTube:
Part 1 (http://www.youtube.com/watch?v=4PSPvHcLnN0)
Part 2 (http://www.youtube.com/watch?v=CtHlxzOXgYs)
Title: Re: IBM AI plays Jeopardy
Post by The_Jeh on Feb 16th, 2011, 1:34am
Hmm... Watson really crushed them yesterday. I wonder how much of it has to do with buzzer reaction time. I know that it is pressing an actual buzzer via a robotic mechanism. Still, it should take a human much more reaction time to see the lights turn off and then press the button by thumb than it would take a computer to receive the go-ahead via circuit and then use the robotic mechanism.

But strictly from the standpoint of how well Watson is answering, I am impressed. True, there were some exceptions, most notably in Final Jeopardy:
http://voices.washingtonpost.com/the-buzz/2011/02/watson_on_jeopardy_tongiht.html
http://www.eweek.com/c/a/IT-Infrastructure/IBMs-Watson-Wins-in-Jeopardy-Needs-Lessons-in-US-Geography-845727/

I'll probably miss Game 2 because of work. Hopefully I'll be able to see it somewhere later.
Title: Re: IBM AI plays Jeopardy
Post by rbarreira on Feb 16th, 2011, 5:05am
Apparently lots of people are complaining that Watson's advantage seems to be more in the button pressing time than in knowing the answers.
Title: Re: IBM AI plays Jeopardy
Post by RonWeasley on Feb 16th, 2011, 7:21am
Even with the button pressing advantage, I'm still impressed with Watson.

One thing bothering me is IBM trying to oversell the technology.  At one point they were saying a Watson-like system could replace doctors for medical diagnosis.  No it can't.  However, it can be very helpful in assisting doctors.  On a related note, IBM seems to be saying the Watson technology is new.  It's not.  It's just bigger and impressively general.  Specialized decision aid systems have been around for about twenty years.  I'm still impressed, but I hate seeing the executives mislead the public.
Title: Re: IBM AI plays Jeopardy
Post by Adanac on Feb 16th, 2011, 9:53am

on 02/16/11 at 07:21:47, RonWeasley wrote:
Even with the button pressing advantage, I'm still impressed with Watson.


I was also very impressed with Watson's performance.  I don't know much about AI but I would have intuitively expected that the Arimaa Challenge would be won before the best Jeopardy contestants were defeated.  And I was surprised to learn that I don't live in a U.S. city  :o

I think this highly publicized event will help Arimaa in the long-run especially if we can win the Arimaa Challenge 9-0 again.  It would be great to tell the world "Brad Rutter and Ken Jennings couldn't stop Watson but 3 Arimaa players trounced bot_whatever 9-0".
Title: Re: IBM AI plays Jeopardy
Post by Fritzlein on Feb 16th, 2011, 12:47pm

on 02/16/11 at 09:53:31, Adanac wrote:
I think this highly publicized event will help Arimaa in the long-run especially if we can win the Arimaa Challenge 9-0 again.  It would be great to tell the world "Brad Rutter and Ken Jennings couldn't stop Watson but 3 Arimaa players trounced bot_whatever 9-0".

I hope this is the public reaction, as opposed to: "IBM could master Arimaa any time they wanted to (just like they did with chess and Jeopardy), but the Arimaa Challenge rules prevent them from competing."

Hopefully, if humans are still way ahead in 2020 when the Arimaa Challenge expires, the community will have grown enough that the question becomes interesting apart from Omar's money.  In that case, people might start building/measuring Arimaa engines on more powerful hardware.
Title: Re: IBM AI plays Jeopardy
Post by mistre on Feb 16th, 2011, 1:34pm
Watched it last night.  While it was neat in concept, the whole contest seems unfair because of the reaction time issue.  Watson can obviously buzz in faster the human participants (I guarantee that they knew at least 80% of the questions based on their previous Jeopardy appearances).

Still, I was impressed with the results.  It just didn't make for a very exciting match.  Maybe tonight will be better.
Title: Re: IBM AI plays Jeopardy
Post by RonWeasley on Feb 17th, 2011, 8:10am

on 02/16/11 at 07:21:47, RonWeasley wrote:
Even with the button pressing advantage, I'm still impressed with Watson.

I'm also impressed with Emma Watson.  She looks very much like the real Hermione.
Title: Re: IBM AI plays Jeopardy
Post by mistre on Feb 17th, 2011, 8:31am
Definitely more entertaining last night. Ken must have been taking button pushing lessons as he was able to buzz in more frequently. Loved his sense of humor at the end to - "I for one welcome our new computer overlords".

In what year will we being saying the same about Arimaa??
Title: Re: IBM AI plays Jeopardy
Post by arimaa_master on Feb 17th, 2011, 9:58am

on 02/17/11 at 08:10:58, RonWeasley wrote:
I'm also impressed with Emma Watson.  She looks very much like the real Hermione.


LOL
Title: Re: IBM AI plays Jeopardy
Post by Fritzlein on Feb 17th, 2011, 10:25am

on 02/17/11 at 08:10:58, RonWeasley wrote:
I'm also impressed with Emma Watson.

It's hard not to be impressed impressed with Emma Watson, unless you're looking with your eyes closed. :o


on 02/16/11 at 09:53:31, Adanac wrote:
I think this highly publicized event will help Arimaa in the long-run[...]

Apparently it has already helped Arimaa in the short run; witness the latest message in the "Say Hello" thread.  :)
Title: Re: IBM AI plays Jeopardy
Post by omar on Feb 17th, 2011, 1:50pm
Wow, IBM has done it again. It feels like 1997 again when I read all the hype in the media about this. No doubt this is a major accomplishment. Although I didn't like the fact that Watson did have some advantages with being given the text of the question immediately and being able to press the buzzer so quickly. It would have been a more interesting contest if Watson had to use speech recognition to get the questions and literally use a mechanical hand to press the buzzer :-)

The technology coming from this project, though is quite interesting and could have immediate applications. Sounds like IBM is planning to use it to assist doctors with diagnosing patients. I hope they also setup a 'Ask Watson' site and allow people to ask it questions.

I think it might even have some long term implications for Arimaa and other games. It shows that one doesn't really have to understand the meaning to come up with a good answer if you have a huge knowledge base available. In the context of game playing, programs that use the conventional tree search approach are essentially trying to understand the meaning of the position and come up with an answer all on their own. The other extreme is when they use an opening or endgame book (essentially a huge knowledge base) to just lookup the response in a brain dead way. But the Watson project suggests that their may be a good middle ground. What if an Arimaa playing program maintained a knowledge base of all the Arimaa games and used a similarity match to identify previous positions similar to the current position and looked at the kind of moves that lead to wins in previous games and picked a similar move available from the current position after doing a quick 2 ply check to make sure the move is not an obvious blunder. Such an approach would probably not be very successful initially compared to the current best bots, but it may eventually be able to reach a higher level of play. This approach to playing would certainly be closer to how humans play.
Title: Re: IBM AI plays Jeopardy
Post by The_Jeh on Feb 17th, 2011, 1:57pm
Alex Trebek: "This is the best opening move in a game of Arimaa."

Watson: "What is...."

:)
Title: Re: IBM AI plays Jeopardy
Post by rbarreira on Feb 17th, 2011, 4:09pm
I think both the human contestants and Watson get the question text immediately, so all that's left to complain about in that regard is possibly that humans are slower at reading text.

But that would be a specious argument too, since Watson must also take some time to analyse the meaning of the question and might even evaluate it several times in a feedback loop with the search algorithms (just a guess). Also, if you give the text later to Watson it starts looking like giving a handicap to favor the humans, reminding us of the Kramnik vs Deep Fritz match...

I do somewhat agree with the "buzzer" concern though, that's probably because I think about Watson's feat as being language analysis rather than having fast mechanical devices (which is of course old technology and therefore shouldn't be the focus here).
Title: Re: IBM AI plays Jeopardy
Post by Janzert on Feb 17th, 2011, 4:51pm
While I certainly agree Watson has an advantage in buzzer pressing response time, I'm not sure it's clear to everyone here that it was indeed pressing a mechanical switch the same as a human contestant does.

Also anyone/thing relying on speech recognition to get the clue from Trebek's reading of it is going to be at a huge disadvantage. Certainly almost all the players are reading it off the displays in the studio, not waiting to listen to the reading. You could argue Watson should have had to OCR it from cameras aimed at the displays too, but that doesn't seem very technically interesting.

Janzert
Title: Re: IBM AI plays Jeopardy
Post by Fritzlein on Feb 17th, 2011, 5:57pm

on 02/17/11 at 16:09:39, rbarreira wrote:
I think both the human contestants and Watson get the question text immediately, so all that's left to complain about in that regard is possibly that humans are slower at reading text.

You are neglecting a certain complication.  In order to make Jeopardy suitable for television, the players are not allowed to hit their buzzers before Alex finishes reading the question out loud.  Therefore, there will be questions for which Ken Jennings knows the answer a full second before Alex is done reading, but he is not allowed to buzz in.  At some point, all the contestants get a signal "it is OK to buzz in now".  Of course, if all three players know the answer and all three have already decided to buzz in, then Watson will win the reaction-time game to buzz in fastest after being told it is OK to buzz in.  So what is being tested is not "how fast can you understand this question and come up with an answer", but rather, "how fast can you physically react to a signal".


Quote:
But that would be a specious argument too, since Watson must also take some time to analyse the meaning of the question and might even evaluate it several times in a feedback loop with the search algorithms (just a guess). Also, if you give the text later to Watson it starts looking like giving a handicap to favor the humans, reminding us of the Kramnik vs Deep Fritz match...

In fact, the rules of Jeopardy handicap the humans by preventing them from buzzing in as soon as they know the answer.  If you removed that handicap, gave all the contestants the question text simultaneously, and let them all buzz in whenever they were ready, then Watson would have come in third place by a mile, because the human champions can parse and understand the question faster.

Did you notice in the second day, in the category "Actor-Director", the humans got every question and Watson got none.  Why?  We could see Watson's thinking flash onto the screen, and Watson every time had the correct answer with a high confidence.  The difference was that the questions were all very short, i.e. they were all just movie titles, so they didn't take long to read.  This removed the handicap of having to wait for Alex to finish reading, so the humans beat Watson to the buzzer every time.

I am sure that some day the reverse would be true.  Probably in a few years Watson could come back with the answer in under a second.  For now, however, Watson would have lost a "fair" match.  It is just a silly quirk of Jeopardy for television that all the contestants are artificially slowed down.
Title: Re: IBM AI plays Jeopardy
Post by mistre on Feb 17th, 2011, 10:35pm
Thanks for clearing that up Fritzlein.  It makes more sense to me now.  Seems to me that the biggest variable to how well the contestants would do was question length then.  Which makes for a very flawed match.  

The only way to make it a fair match (without completely changing the rules of Jeopardy) would be to time the reaction time of Watson and tailor each and every question to take the same exact amount of time for Alex to read - so as to synchronize the time that the contestants would buzz in if they knew the answer.  
Title: Re: IBM AI plays Jeopardy
Post by omar on Feb 18th, 2011, 8:31am

on 02/17/11 at 17:57:11, Fritzlein wrote:
Did you notice in the second day, in the category "Actor-Director", the humans got every question and Watson got none.  Why?  We could see Watson's thinking flash onto the screen, and Watson every time had the correct answer with a high confidence.  The difference was that the questions were all very short, i.e. they were all just movie titles, so they didn't take long to read.  This removed the handicap of having to wait for Alex to finish reading, so the humans beat Watson to the buzzer every time.


Good observation Karl. I was thinking Watson was just not good at that topic, but your explanation makes more sense. While the rest of the world is happily accepting the computers victory, you've recognized the flaws in the Jeopardy game rules :-)

What could be done to make Jeopardy more fair in matches between humans and computers?
Title: Re: IBM AI plays Jeopardy
Post by megajester on Feb 18th, 2011, 8:59am

on 02/18/11 at 08:31:27, omar wrote:
What could be done to make Jeopardy more fair in matches between humans and computers?

Put all contestants in separate rooms, ask the question, give humans and computers x seconds to formulate an answer and make them give it within 1 second of the buzzer sounding. Provide the computer with the question (or bits of it) in text form only once the humans have heard it (or heard enough to understand it).

EDIT: I only know Jeopardy from Groundhog Day so I hope I'm not making a complete idiot of myself. Still, if the whole point of the exercise is to see who can give the right answer fastest, then you have to make sure everybody understands the question at the same time and can buzz at the same speed when they have the answer. Giving the question to the computer only once the players have heard it sounds like a possibility for the first consideration. For the second, you could delay computer buzzing in line with human response times, eg. the amount of time it takes, say, a driver to push the break pedal after a hazard appears.
Title: Re: IBM AI plays Jeopardy
Post by Sconibulus on Feb 18th, 2011, 9:33am
I'd think the easiest solution would be to permit early buzzing, and possibly take away the player's ability to read the whole question in advance. That's the way it was done in the vaguely jeapordish team trivia competitions in high school. In that case, you could buzz in whenever you wanted and try to answer, but you only lost points on an incorrect answer if you buzzed before the question had been entirely read... I suppose that is changing the game rather drastically though, and it would probably result in less game-time per episode aired unless the board size was increased.
Title: Re: IBM AI plays Jeopardy
Post by Fritzlein on Feb 18th, 2011, 10:13am

on 02/18/11 at 08:31:27, omar wrote:
What could be done to make Jeopardy more fair in matches between humans and computers?

Well, "fair" is a really hard concept once you understand what each side can do well.  For example, the question-writers would probably notice before to long that certain kinds of questions are more difficult for the computer to get correct.  If they increased the percentage of such questions, the audience probably wouldn't notice, but it wouldn't be fair, would it?  Also measuring buzz-in-reaction-time isn't "unfair", because it is the same for all players.  It's just something machines happen to be good at.

I think I am less interested in a fair match than I am interested in an interesting match.  The buzz-in-reaction-time issue doesn't bug me by virtue of being unfair, it bugs me by virtue of being obvious who will win.  Yes, we know it takes a human 1/8th of a second to buzz in and a computer 1/100th of a second.  Boring.  Change the rules to make it unboring.

That said, on some level I respect the organizers for changing Jeopardy as little as possible to let Watson play.  Whether the playing field of Jeopardy is naturally tipped to humans or to computers, at least they didn't change how much it is naturally tipped.  :)
Title: Re: IBM AI plays Jeopardy
Post by megajester on Feb 18th, 2011, 10:20am
Now why does this all remind me of the Kasparov v. Deep Blue match?...

New theory:

IBM hasn't taken on Arimaa yet because they're only interested in stuff they're guaranteed to win.

As indeed any hard-nosed businessman should be.

All IBM want to do is sell their top-notch hardware. The whole point of the Arimaa Challenge is to win using something other than top-notch hardware. Therefore if IBM ever take up the Arimaa Challenge I'll eat my hat.

I think more attention could be drawn to Arimaa (and from more worthy quarters than IBM) if it could be somehow demonstrated that computers by rights should be able to win the Arimaa Challenge. Which you can't do... without winning the Arimaa Challenge. Or can you?
Title: Re: IBM AI plays Jeopardy
Post by Fritzlein on Feb 18th, 2011, 10:27am

on 02/18/11 at 10:20:07, megajester wrote:
IBM hasn't taken on Arimaa yet because they're only interested in stuff they're guaranteed to win.

Heh, if that were so, then IBM wouldn't have taken on either chess or Jeopardy.  Both were very risky and uncertain.  What IBM really wants is good (free) publicity.  But good publicity comes from doing something that millions of people think can't be done.  It is hard to find something that millions of people think can't be done and you are guaranteed to win.

The trouble with Arimaa is that there aren't even a million people who know what it is, never mind a million people who think it can't be ruled by a computer.  IBM will ignore us, but not primarily because Arimaa is an uninteresting challenge.  So first we have to get millions of people hooked on playing Arimaa.  :-)
Title: Re: IBM AI plays Jeopardy
Post by megajester on Feb 18th, 2011, 10:50am

on 02/18/11 at 10:27:57, Fritzlein wrote:
Heh, if that were so, then IBM wouldn't have taken on either chess or Jeopardy.

OK you're right that they weren't guaranteed to win and you know the ins and outs better than I do. But I'm sure you'll grant me that no board of directors would have approved either without reasonable assurances from their experts that they were do-able.

Also, IBM sells hardware. By definition, the Arimaa Challenge removes all potential for showcasing hardware.

As you say, to attract any big players to the Challenge, we need to get millions of people playing Arimaa. :) And I'm hopeful that is possible. But if it does happen, I suspect they will all be watching because of their love of Arimaa, not their interest in AI. As I think you allude to in your book, and as Tiago Luchini over at BGG (http://boardgamegeek.com/thread/372482/arimaa-or-wow-theyve-fixed-chess) expresses in his own way, to promote Arimaa as just "chess tweaked so humans can still beat computers" sells the game short. It's a rich gaming experience that I know we all share, and I personally still have trouble putting into words. And I think it's that unique experience that will guarantee not only the prestige of the Arimaa Challenge, but the future of Arimaa.
Title: Re: IBM AI plays Jeopardy
Post by The_Jeh on Feb 18th, 2011, 11:46am

on 02/18/11 at 10:27:57, Fritzlein wrote:
Heh, if that were so, then IBM wouldn't have taken on either chess or Jeopardy.  Both were very risky and uncertain.  What IBM really wants is good (free) publicity.  But good publicity comes from doing something that millions of people think can't be done.  It is hard to find something that millions of people think can't be done and you are guaranteed to win.

The trouble with Arimaa is that there aren't even a million people who know what it is, never mind a million people who think it can't be ruled by a computer.  IBM will ignore us, but not primarily because Arimaa is an uninteresting challenge.  So first we have to get millions of people hooked on playing Arimaa.  :-)


Of course IBM wants publicity, but I think the nature of the Jeopardy challenge as opposed to the Arimaa challenge has as much to do with IBM's choice as the size of the audience they will get.  I'm not saying that the AI advances needed to play Arimaa effectively will turn out to be less usefully applicable (and hence less marketable) than those needed to play Jeopardy. I am saying that investors probably will see it that way, especially since chess has already been done. IBM has not proven their innovative prowess alone this time. Rather, they have demonstrated something that is almost an actual marketable product.
Title: Re: IBM AI plays Jeopardy
Post by rbarreira on Feb 24th, 2011, 3:26am
Kasparov has made some comments (http://www.theatlantic.com/technology/archive/2011/02/exclusive-garry-kasparov-on-ibms-watson/71584/) on Watson. On the whole his comments don't seem unreasonable, but it seems he still hasn't given up the idea that humans are superior to computers at chess... (I highlighted that part)


Quote:
       * A convincing victory under strict parameters, and if we stay within those limits Watson can be seen as an incremental advance in how well machines understand human language. But if you put the questions from the show into Google, you also get good answers, even better ones if you simplify the questions. To me, this means Watson is doing good job of breaking language down into points of data it can mine very quickly, and that it does it slightly better than Google does against the entire Internet.

       * Much like how computers play chess, reducing the algorithm into "crunchable" elements can simulate the way humans do things in the result even though the computer's method is entirely different. If the result—the chess move, the Jeopardy answer—is all that matters, it's a success. If how the result is achieved matters more, I'm not so sure. For example, Deep Blue had no real impact on chess or science despite the hype surrounding its sporting achievement in defeating me. If Watson's skills can be translated into something useful, something groundbreaking, that is the test. If all it can do is beat humans on a game show Watson is just a passing entertainment akin to the wind-up automata of the 18th century.

       * My concern about its utility, and I read they would like it to answer medical questions, is that Watson's performance reminded me of chess computers. They play fantastically well in maybe 90% of positions, but there is a selection of positions they do not understand at all. Worse, by definition they do not understand what they do not understand and so cannot avoid them. A strong human Jeopardy! player, or a human doctor, may get the answer wrong, but he is unlikely to make a huge blunder or category error—at least not without being aware of his own doubts. We are also good at judging our own level of certainty. A computer can simulate this by an artificial confidence measurement, but I would not like to be the patient who discovers the medical equivalent of answering "Toronto" in the "US Cities" category, as Watson did.

       * I would not like to downplay the Watson team's achievement, because clearly they did something most did not yet believe possible. And IBM can be lauded for these experiments. I would only like to wait and see if there is anything for Watson beyond Jeopardy!. These contests attract the popular imagination, but it is possible that by defining the goals so narrowly they are aiming too low and thereby limit the possibilities of their creations.
Title: Re: IBM AI plays Jeopardy
Post by ddyer on Feb 24th, 2011, 1:39pm

on 02/18/11 at 08:31:27, omar wrote:
Good observation Karl. I was thinking Watson was just not good at that topic, but your explanation makes more sense. While the rest of the world is happily accepting the computers victory, you've recognized the flaws in the Jeopardy game rules :-)

What could be done to make Jeopardy more fair in matches between humans and computers?


I don't find this plausible.  Assuming humans and Watson are both "ready" before the allowed time to buzz in, Watson has nanosecond awareness of time and should be able to beat the humans every time.

In any case, the amazing thing about Watson is that Jeopordy becomes mostly a question of reflexes.  To be that good at simulating natural language comprehension is awesome.
Title: Re: IBM AI plays Jeopardy
Post by The_Jeh on Feb 24th, 2011, 2:06pm

on 02/24/11 at 13:39:29, ddyer wrote:
 To be that good at simulating natural language comprehension is awesome.


"Simulating" is a good word. Watson is impressive with natural language input, but I don't know if I'd choose the verb "understand." If it can be made to "understand" one natural language, then it should be able to be made to "understand" two natural languages. In that case, IBM wouldn't be far away from translating between the two natural languages fluently.
Title: Re: IBM AI plays Jeopardy
Post by Fritzlein on Feb 24th, 2011, 5:08pm

on 02/24/11 at 13:39:29, ddyer wrote:
I don't find this plausible.  Assuming humans and Watson are both "ready" before the allowed time to buzz in, Watson has nanosecond awareness of time and should be able to beat the humans every time.

And in fact that is what happened.  The only times that humans beat Watson to the buzzer were when (A) Watson wasn't sure of the answer and thus held off or (B) Watson wasn't "ready" at the time Alex had finished reading the question, but the humans were.

How does that not fit with what you observed?
Title: Re: IBM AI plays Jeopardy
Post by Fritzlein on Mar 1st, 2011, 11:45pm
Watson goes on tour, is beaten by congressman:
http://www.cleveland.com/newsflash/index.ssf/story/us-congressman-tops-jeopardy-computer-watson/b8984784c9d4454db01523b8d99b6d8e
Title: Re: IBM AI plays Jeopardy
Post by omar on Mar 3rd, 2011, 11:23pm
Wow, that's quite amazing. Holt must have some pretty good reaction time :-)
Title: Re: IBM AI plays Jeopardy
Post by Fritzlein on Mar 4th, 2011, 9:35am
Stephen Colbert had a funny commentary on the game: "Does this mean the Democrats have achieved self-awareness?"
Title: Re: IBM AI plays Jeopardy
Post by Arimabuff on Mar 21st, 2011, 4:45am

on 03/04/11 at 09:35:20, Fritzlein wrote:
Stephen Colbert had a funny commentary on the game: "Does this mean the Democrats have achieved self-awareness?"

I am guessing Stephen Colbert is a Republican.
Title: Re: IBM AI plays Jeopardy
Post by qswanger on Mar 21st, 2011, 7:56am

on 03/21/11 at 04:45:19, Arimabuff wrote:
I am guessing Stephen Colbert is a Republican.


I'm guessing you've never heard The Colbert Report:
http://en.wikipedia.org/wiki/Stephen_Colbert
Title: Re: IBM AI plays Jeopardy
Post by Arimabuff on Mar 21st, 2011, 10:24am

on 03/21/11 at 07:56:26, qswanger wrote:
I'm guessing you've never heard The Colbert Report:
http://en.wikipedia.org/wiki/Stephen_Colbert

I was thinking of the sketch where Bop Hope compares Democrats to zombies. Here it is:

http://www.youtube.com/watch?v=LNfhsUcyld0


Arimaa Forum » Powered by YaBB 1 Gold - SP 1.3.1!
YaBB © 2000-2003. All Rights Reserved.