Welcome, Guest. Please Login or Register.
Mar 29th, 2024, 4:04am

Home Home Help Help Search Search Members Members Login Login Register Register
Arimaa Forum « 2007 Arimaa Challenge Match »


   Arimaa Forum
   Arimaa
   Events
(Moderator: supersamu)
   2007 Arimaa Challenge Match
« Previous topic | Next topic »
Pages: 1 2  Reply Reply Notify of replies Notify of replies Send Topic Send Topic Print Print
   Author  Topic: 2007 Arimaa Challenge Match  (Read 4002 times)
Fritzlein
Forum Guru
*****



Arimaa player #706

   
Email

Gender: male
Posts: 5928
Re: 2007 Arimaa Challenge Match
« Reply #15 on: Feb 14th, 2007, 1:06pm »
Quote Quote Modify Modify

For the record, I was afraid the qualifying phase could result in catastrophe.  Bomb is clearly the stronger bot, but there seemed to be enough randomness in the process that Zombie could win "accidentally".  However, with ten game pairs tied and the two decisive pairs won by Bomb, the bot that "should have" won actually did win by a comfortable margin.  We'll see what happens in the future if the two bots are closer in strength.
 
I'm happy that eight different people played at least one pair.  I expect that next year the number will be higher as folks get used to the concept, which hopefully will make the result more and more reliable.
 
Most importantly, however, the qualifying games fully served the purpose of not letting a new bot sneak up on us.  The pairs where humans won both games, although they didn't provide discrimination, were proof of concept of various ways of winning.  For example, even supposing we had had no prior experience against Zombie, Belbo's awe-inspiring 28-move immobilization would be a demonstration that the Challenge defenders could draw on.  Similarly for chessandgo's 18-move win by goal.  Any time there is a new bot on the scene, the "qualifying phase" will be critical as the first leg of humanity's defense.
IP Logged

IdahoEv
Forum Guru
*****



Arimaa player #1753

   


Gender: male
Posts: 405
Re: 2007 Arimaa Challenge Match
« Reply #16 on: Feb 14th, 2007, 8:11pm »
Quote Quote Modify Modify

I think, frankly, there should be more time between the WCC and the challenge to allow humans -- including the defenders -- to play against the bots.
 
Even with this system, there is some risk of a programmer "sneaking up" on the humans, by developing a new strategy or approach that was not overall better, but was merely surprising.
 
Imagine some developer manages this in the 2010 challenge, and wins the prize.  And then within a week of study afterwards, the defenders had reliably figured out how to beat the bot.   Did the bot really deserve the prize that year?   It seems to be the prize should be given to the first bot that can consistently beat top players, not just one that sneaks up.
 
Yes, this means that the humans have an opportunity to learn from their early losses to the bot.   But, it also means the bot has an opportunity to learn from its losses as well.  And, it seems to me, any bot likely to truly win at this game is going to need to do just that anyway.
 
I would put the challenge something like two full months after the CC, and let everyone including the defenders play against the bots.
« Last Edit: Feb 14th, 2007, 8:12pm by IdahoEv » IP Logged
IdahoEv
Forum Guru
*****



Arimaa player #1753

   


Gender: male
Posts: 405
Re: 2007 Arimaa Challenge Match
« Reply #17 on: Feb 14th, 2007, 8:19pm »
Quote Quote Modify Modify

on Feb 14th, 2007, 1:06pm, Fritzlein wrote:
For the record, I was afraid the qualifying phase could result in catastrophe.

 
Define "catastrophe":  Zombie wouldn't have beaten the defenders either.    
 
But I see your point.  While at this point it is fairly meaningless which bot plays against the humans, there were an awful lot of timeout games going into the "score" on either side.   Unfortunately, I don't see an answer to that.
IP Logged
Fritzlein
Forum Guru
*****



Arimaa player #706

   
Email

Gender: male
Posts: 5928
Re: 2007 Arimaa Challenge Match
« Reply #18 on: Feb 14th, 2007, 9:22pm »
Quote Quote Modify Modify

on Feb 14th, 2007, 8:19pm, IdahoEv wrote:
Define "catastrophe":  Zombie wouldn't have beaten the defenders either.

By catastrophe I meant that the clearly weaker bot would win by luck.  The closer the two bots are in strength, the less it matters which one wins the qualifying, and in fact it is reasonable that a bot which is slightly weaker head-to-head would be slightly stronger against humans.  
IP Logged

Fritzlein
Forum Guru
*****



Arimaa player #706

   
Email

Gender: male
Posts: 5928
Re: 2007 Arimaa Challenge Match
« Reply #19 on: Feb 14th, 2007, 10:56pm »
Quote Quote Modify Modify

on Feb 14th, 2007, 8:11pm, IdahoEv wrote:
Imagine some developer manages this in the 2010 challenge, and wins the prize.  And then within a week of study afterwards, the defenders had reliably figured out how to beat the bot.   Did the bot really deserve the prize that year?  It seems to be the prize should be given to the first bot that can consistently beat top players, not just one that sneaks up.

I agree that the Arimaa Challenge could be won by software that was not absolutely dominant over humans.
 
There are lots of ways to measure dominance.  With ever-increasing hardware speeds, it is tempting to settle on a very high standard, such as being able to beat all human players with greater than 50% probability, not only today, but for all time, without further developer intervention to fix bugs and add new strategic knowledge that humans may pick up in the mean time.
 
I certainly don't think we need such a high standard to say that computers have passed up humanity at playing Arimaa, but let's say we really do want the Challenge to measure this standard.  My hunch is that software which can beat three top humans each two games out of three (after having been exposed to community play for two weeks) must be very close, even if it isn't all the way there.  If humans soon discover a reliable way to beat such software, it is probably due to a minor quirk that can soon be fixed rather than seriously flawed strategy which worked only due to surprise.  Our understanding of Arimaa is probably mature enough that there isn't some strategy trick nobody has thought of yet, which pushes a bot past us only temporarily without heralding the end of human dominance.  In all likelihood, the error of the current format would only be in awarding the prize one year too soon.
 
But this isn't the main reason I oppose making the Challenge tougher to win.  My main issue would be that it isn't fair to keep moving the goal posts.  When Omar first put out the Challenge the bot was only required to win a match against one top human.  In the third year he changed it to require the bot to beat each of three humans.  This moved back the goalposts significantly from the original requirement.
 
Suppose there really were a dominant bot that could beat all humans 60% of the time (and obviously win even more often against lower-ranked humans).  The chance that it would win an eight-game match against one top human would be 59.4%, even with 4-4 ties going to the human.  But this same dominant computer would only have a 27.2% chance of winning all three mini-matches against three top humans.  Probably one human would get lucky and win twice even if the computer had a 60% chance to win each individual game.
 
In other words, Omar has already moved the goal posts once, and made it so that a bot that is clearly the best player in the world could have a good chance of not winning the Challenge prize.  I believe Omar when he says that the current goal line is closer to what he had in mind all along, but it introduces a significant chance that the Challenge prize will be awarded too late, an error even by the stringent standard of dominance first outlined.
 
Furthermore, the fact that there is a qualifying round at all has toughened the requirement on bots.  In 2005 developers only had to have their bots play at least six games total against at least three different humans, and the strength of the humans was not specified.  For a bot to be open to play for two weeks against all humans (up to four games each) allows much more exposure, and much more probing for weaknesses.  (To be honest, the developers probably don't mind so much since it takes the responsibility of finding human opponents off their shoulders, but for a developer playing purely to win, winning got harder.)
 
If we now extend the qualifying period to two months instead of two weeks and also allow the actual defenders to play as many games as they like against the bots, then it will just appear to be part of the general trend of making the Challenge rules more difficult every year.
 
Yes, there is always a case to be made that, until a program can do X, it isn't totally dominant.  Chess developers suffered for years from man vs. machine competitions taking place under rules more and more favorable to humans.  Developers could be excused for thinking that no matter how good their programs got, humans would find a way to change the rules once again and say that programs aren't really champs.  
 
I feel this is exactly why we have to guard against making the Challenge harder and harder to meet.  Omar should pick a reasonable middle ground that tries both to disallow sneak attacks and to allow a program that really is the best to claim the prize.  Yes, we are still working out the kinks and finalizing the rules, but as we work out the details, the difficulty should stay basically constant and not increase every year.
« Last Edit: Feb 15th, 2007, 8:13pm by Fritzlein » IP Logged

PMertens
Forum Guru
*****



Arimaa player #692

   
WWW

Gender: male
Posts: 437
Re: 2007 Arimaa Challenge Match
« Reply #20 on: Feb 15th, 2007, 3:05pm »
Quote Quote Modify Modify

If a bot should be able to beat the defenders only to be shown to have a weakness two months later then it certainly still should get the prize imho.
It is very likely that in a fixed performance bot somebody will eventually find a weakness ... but I do not think the point of the exercise is to create a flawless bot, just one that proves that humans will eventually be outdated no matter how much they try to cheat Wink
 
At the moment the bots are so incredible far away from being even close to a threat to the (top)humans, that anything capable of beating 50% would be an absolute breakthrough.
No need to move goalposts since once that happens we will all know that it is only a matter of (short) time until we lost this game to the comps.
IP Logged
jdb
Forum Guru
*****



Arimaa player #214

   


Gender: male
Posts: 682
Re: 2007 Arimaa Challenge Match
« Reply #21 on: Feb 15th, 2007, 6:00pm »
Quote Quote Modify Modify

Quote:
...Our understanding of Arimaa is probably mature enough that there isn't some strategy trick nobody has thought of yet, ...

 
 Shocked
IP Logged
Fritzlein
Forum Guru
*****



Arimaa player #706

   
Email

Gender: male
Posts: 5928
Re: 2007 Arimaa Challenge Match
« Reply #22 on: Feb 15th, 2007, 8:06pm »
Quote Quote Modify Modify

Heh, JDB, you only quoted half my sentence.  I actually expect there are tons of Arimaa strategies we haven't thought of yet.  If a bot surpasses humans, it will probably be by better execution of simple strategies than by bringing new strategies to the table, in my humble opinion.  However, IF a bot should chance to win by bringing a new strategy to the table, THEN it will mark the end of human dominance no matter how we try to adapt.  I'm not saying that we can't be surprised by anything new, just that if we are beaten by X, whatever X is, it is probably too late for X to be a cheap trick that works only one year, after which humans regain ascendancy.
IP Logged

jdb
Forum Guru
*****



Arimaa player #214

   


Gender: male
Posts: 682
Re: 2007 Arimaa Challenge Match
« Reply #23 on: Feb 16th, 2007, 2:03pm »
Quote Quote Modify Modify

Hi Fritz,  
 
I agree with your statement. Your quote was just too juicy for me to ignore, even if it was taken out of context.   Smiley
IP Logged
Belteshazzar
Forum Guru
*****



Arimaa player #5094

   


Gender: male
Posts: 108
Re: 2007 Arimaa Challenge Match
« Reply #24 on: Feb 2nd, 2019, 4:47pm »
Quote Quote Modify Modify

on Feb 14th, 2007, 10:56pm, Fritzlein wrote:
I certainly don't think we need such a high standard to say that computers have passed up humanity at playing Arimaa, but let's say we really do want the Challenge to measure this standard.  My hunch is that software which can beat three top humans each two games out of three (after having been exposed to community play for two weeks) must be very close, even if it isn't all the way there.  If humans soon discover a reliable way to beat such software, it is probably due to a minor quirk that can soon be fixed rather than seriously flawed strategy which worked only due to surprise.  Our understanding of Arimaa is probably mature enough that there isn't some strategy trick nobody has thought of yet, which pushes a bot past us only temporarily without heralding the end of human dominance.  In all likelihood, the error of the current format would only be in awarding the prize one year too soon.

 
Looks like this was largely correct.
« Last Edit: Feb 2nd, 2019, 4:49pm by Belteshazzar » IP Logged
Pages: 1 2  Reply Reply Notify of replies Notify of replies Send Topic Send Topic Print Print

« Previous topic | Next topic »

Arimaa Forum » Powered by YaBB 1 Gold - SP 1.3.1!
YaBB © 2000-2003. All Rights Reserved.