Arimaa Forum
        (http://arimaa.com/arimaa/forum/cgi/YaBB.cgi)
      

        Arimaa >> Events >> 2008 Arimaa Challenge
        
(Message started by: RonWeasley on Mar 13th, 2008, 7:36pm)
Title: 2008 Arimaa Challenge
Post by RonWeasley on Mar 13th, 2008, 7:36pm
As match director, I am pleased to announce the bot challengers and human defenders for the 2008 Arimaa Challenge.

First, 2008 champion bot_Bomb2005CC and newcomer bot_sharp will be made available for non-defender humans to play.  Then the bot with the best record in these games will earn the right to challenge the selected defenders, each to a best-of-three game match.

This year the defenders are 2007 World Champion Jean Daligault (chessandgo), Greg Magne (Adanac), and Mark Mistretta (mistre).  The alternate is Omar Syed (omar).  Thank you for defending the honor of all humanity!

Please visit the Arimaa Challenge page for complete match rules.  Good luck to the challengers and to the defenders!
Title: Re: 2008 Arimaa Challenge
Post by omar on Mar 15th, 2008, 1:40am
Open season on the two best bots is about to begin.
The result of how the bots do against humans will
determine which bot goes on to play in the challenge
match. The bots will be available to play for two weeks beginning March 16.

Use this link to play against the bots.

http://arimaa.com/arimaa/wcc/2008/playBestBots.cgi

The human defenders and the bot developers are not allowed to play these bots during this time.
Title: Re: 2008 Arimaa Challenge
Post by Fritzlein on Mar 15th, 2008, 6:20pm
Omar and Ron (and everyone), how would you feel if I played the qualifying games at handicap?  I expect that if I play standard games, I will win two against sharp and win two against Bomb.  This provides no information in distinguishing which bot is more deserving of being in the finals.  However, if I were to play all four of the games at a camel handicap, I would probably be as likely to lose each game as to win.  This might actually give some differentiation between the two bots.

I can see a case against playing handicap games.  Even if a handicap is "fair" in that I use the same handicap against each bot, the information it provides is not exactly what we want to know.  We aren't interested in which bot is best against humans with an extra camel, we are interested in which bot is best against humans at real games.

Which do we want to have: no useful information whatsoever, or some information that isn't exactly what we wanted to know?
Title: Re: 2008 Arimaa Challenge
Post by Fritzlein on Mar 15th, 2008, 7:12pm
Hi, Omar.  The challenge bots are starting at the wrong time control.  I just clicked on the Play Bots link and got a game against Bomb at 60s/120s/75/0/4h/4m.  Meanwhile aaaa is playing sharp at 2/2/100/2/0.  We need that to be fixed, and need a ruling as to whether sharp vs. aaaa counts in the standings.

[EDIT]

Also the pairing code complained when aaaa tried to play in unrated mode, so he turned unrated mode off and tried again, but the game was unrated anyway!
Title: Re: 2008 Arimaa Challenge
Post by arimaa_master on Mar 16th, 2008, 5:22am

on 03/15/08 at 18:20:19, Fritzlein wrote:
Omar and Ron (and everyone), how would you feel if I played the qualifying games at handicap?  I expect that if I play standard games, I will win two against sharp and win two against Bomb.  This provides no information in distinguishing which bot is more deserving of being in the finals.  However, if I were to play all four of the games at a camel handicap, I would probably be as likely to lose each game as to win.  This might actually give some differentiation between the two bots.

I can see a case against playing handicap games.  Even if a handicap is "fair" in that I use the same handicap against each bot, the information it provides is not exactly what we want to know.  We aren't interested in which bot is best against humans with an extra camel, we are interested in which bot is best against humans at real games.

Which do we want to have: no useful information whatsoever, or some information that isn't exactly what we wanted to know?


Yeah, I was thinking about the same (but probably giving only a cat handicap myself :)).

Generally I would like to see such handicap games, it will be at least very interesting. (And why only camel handicap? I think you can give camel + cat handicap without much problems too :)).
Title: Re: 2008 Arimaa Challenge
Post by omar on Mar 16th, 2008, 8:46am

on 03/15/08 at 19:12:14, Fritzlein wrote:
Hi, Omar.  The challenge bots are starting at the wrong time control.  I just clicked on the Play Bots link and got a game against Bomb at 60s/120s/75/0/4h/4m.  Meanwhile aaaa is playing sharp at 2/2/100/2/0.  We need that to be fixed, and need a ruling as to whether sharp vs. aaaa counts in the standings.

[EDIT]

Also the pairing code complained when aaaa tried to play in unrated mode, so he turned unrated mode off and tried again, but the game was unrated anyway!


Oops, the bot config files had the wrong settings. I've fixed it now. Since the games were not rated, they are not picked up by the script which keeps score:

http://arimaa.com/arimaa/wcc/2008/showBestBot.cgi
Title: Re: 2008 Arimaa Challenge
Post by Fritzlein on Mar 16th, 2008, 11:55am

on 03/16/08 at 08:46:24, omar wrote:
http://arimaa.com/arimaa/wcc/2008/showBestBot.cgi

It looks like this link is working, now, although it excludes the two unrated games.
Title: Re: 2008 Arimaa Challenge
Post by RonWeasley on Mar 16th, 2008, 1:13pm
After some discussion with several players, I'm going to rule that the games played under the wrong time control do not count as qualified games, since the rules specify conditions for the time control.  Thank you to those who are playing qualified games and apologies to those whose games have been disqualified.

TD
Title: Re: 2008 Arimaa Challenge
Post by aaaa on Mar 16th, 2008, 2:39pm
I am currently unable to play a qualifier (getting a "Bot Not Started" page), probably because bot_Bomb2005CC is already in a game and the system doesn't allow me to play against bot_sharp again before I've equalized with Bomb. Can this restriction be lifted such that it merely prefers an equalizing game?
Title: Re: 2008 Arimaa Challenge
Post by omar on Mar 17th, 2008, 12:23pm
When there is no preference for which bot you need to play, the script just picks one at random and tries to start it on a server that i not already busy. Thus it is possible to have bomb or sharp running twice, one on each server. If both servers are busy, it will report it and not try to start the bot.

In this case it seems like it tried to start the bot, but it did not start. Im not sure what caused the bot to not start this time, but if it continues please let me know.
Title: Re: 2008 Arimaa Challenge
Post by 99of9 on Mar 17th, 2008, 4:45pm
Don't resignations count as a win for the bot?

Retrovirus naturally resigned when he lost his elephant, but on the score page, it's not showing up as a win for the bot.
Title: Re: 2008 Arimaa Challenge
Post by Janzert on Mar 17th, 2008, 4:49pm
I believe the corresponding game with the other bot has to be played before the score is counted. At least I thought I saw that happen with aaaa's game too.

Janzert
Title: Re: 2008 Arimaa Challenge
Post by aaaa on Mar 17th, 2008, 4:50pm
Sure it does. Shift-reload the page.
Title: Re: 2008 Arimaa Challenge
Post by Fritzlein on Mar 17th, 2008, 7:40pm

on 03/16/08 at 14:39:52, aaaa wrote:
I am currently unable to play a qualifier (getting a "Bot Not Started" page), probably because bot_Bomb2005CC is already in a game and the system doesn't allow me to play against bot_sharp again before I've equalized with Bomb.

Now I'm getting the same error, even though only The_Jeh is playing Bomb, so the other server should be free.
Title: Re: 2008 Arimaa Challenge
Post by mistre on Mar 18th, 2008, 8:32am

on 03/17/08 at 16:45:22, 99of9 wrote:
Don't resignations count as a win for the bot?

Retrovirus naturally resigned when he lost his elephant, but on the score page, it's not showing up as a win for the bot.


It appears that resignations do not count as a win for the bot.  

After thinking that Bomb2005CC was going to walk away this, ChrisB tightened things up by beating Bomb and losing to Sharp.

Go Sharp!
Title: Re: 2008 Arimaa Challenge
Post by The_Jeh on Mar 18th, 2008, 9:04am

on 03/17/08 at 16:45:22, 99of9 wrote:
Don't resignations count as a win for the bot?

Retrovirus naturally resigned when he lost his elephant, but on the score page, it's not showing up as a win for the bot.


No wins count unless the other bot is also played using the same colors. But, I don't know whether resignations are any different.
Title: Re: 2008 Arimaa Challenge
Post by mistre on Mar 18th, 2008, 11:11am

on 03/18/08 at 09:04:22, The_Jeh wrote:
No wins count unless the other bot is also played using the same colors. But, I don't know whether resignations are any different.


You are right, it just so happens that all of the resigning players have yet to play the other bot.  So unless they do, those games won't count.
Title: Re: 2008 Arimaa Challenge
Post by omar on Mar 18th, 2008, 2:24pm

on 03/17/08 at 19:40:49, Fritzlein wrote:
Now I'm getting the same error, even though only The_Jeh is playing Bomb, so the other server should be free.


Thanks for posting the symptoms; it helped me trace the problem to the code for picking a free server. Should be fixed now.
Title: Re: 2008 Arimaa Challenge
Post by Arimabuff on Mar 18th, 2008, 4:10pm
The grid for the results of "Play the Challenge Bots" is out of whack.
Title: Re: 2008 Arimaa Challenge
Post by omar on Mar 18th, 2008, 6:07pm

on 03/18/08 at 16:10:26, Arimabuff wrote:
The grid for the results of "Play the Challenge Bots" is out of whack.


I shuffled the columns a bit.
Title: Re: 2008 Arimaa Challenge
Post by The_Jeh on Mar 18th, 2008, 8:19pm
I noticed I couldn't start Bomb2005CC from the bot ladder. We can't play it outside the quad during this event?
Title: Re: 2008 Arimaa Challenge
Post by Fritzlein on Mar 19th, 2008, 8:58am
It's kind of a silly formality, given that everyone knows Bomb hasn't changed from last year, but the idea is that it isn't fair to challenge bots to expose them to general play.  Someone might experiment endlessly to find a winning formula.  Admittedly, in this case people already have done so, but in general it gives the bot a fighting chance in the Challenge to limit its exposure beforehand.

I think in this case disabling Bomb is more a practical convenience of knowing which games are qualifying games and which games are casual games.  With Bomb disabled in the ladder, all Bomb games in the database within a certain time range are automatically qualifying games.
Title: Re: 2008 Arimaa Challenge
Post by Arimabuff on Mar 19th, 2008, 11:25am

on 03/19/08 at 08:58:56, Fritzlein wrote:
It's kind of a silly formality, given that everyone knows Bomb hasn't changed from last year, but the idea is that it isn't fair to challenge bots to expose them to general play.  Someone might experiment endlessly to find a winning formula.  Admittedly, in this case people already have done so, but in general it gives the bot a fighting chance in the Challenge to limit its exposure beforehand.

I think in this case disabling Bomb is more a practical convenience of knowing which games are qualifying games and which games are casual games.  With Bomb disabled in the ladder, all Bomb games in the database within a certain time range are automatically qualifying games.

Wouldn't it be simpler to put the numbers of the qualifying games into a list?
Title: Re: 2008 Arimaa Challenge
Post by Fritzlein on Mar 19th, 2008, 2:30pm

on 03/19/08 at 11:25:55, Arimabuff wrote:
Wouldn't it be simpler to put the numbers of the qualifying games into a list?

To tell the truth, I don't know how Omar does it.  I should let him answer the questions.
Title: Re: 2008 Arimaa Challenge
Post by omar on Mar 20th, 2008, 1:33am
Yes, I disabled the regular Bomb2005CC temporarily. Otherwise there is no way for the score keeping script to know if you played the one on the bot ladder or the one in the challenge playoff.

I guess, I could enter the game numbers manually into a list, but I wouldn't trust myself to do it in a timely manner.

Sorry for the inconvenience of not being able to play Bomb2005CC on the bot ladder. But this should not hinder ones progress on the bot ladder, because if you beat the challenge playoff version of Bomb it will still count towards the bot ladder. If you don't play Sharp, it won't really hurt things for the playoffs, though it would be nice if you did.
Title: Re: 2008 Arimaa Challenge
Post by Fritzlein on Mar 29th, 2008, 7:32pm
If I'm not mistaken, the qualifying phase just ended.  Congratulations to Bomb for a convincing victory in the playoff as well as in the Computer Championship.

I'm very pleased at how well the qualifying phase worked.  In 2007 there were only 12 pairs of games, and of these 10 were tied.  Only two game pairs were split, and Bomb won by the paltry score of 2-0.  Clearly Bomb was the better bot, but the qualifying didn't provide a whole lot of discrimination.

This year there were 16 pairs of games, and only 9 of those were tied.  A whopping seven pairs of games were split, and Bomb won by the much more persuasive score of 6-1.  If the trend continues, the qualifying phase will get better and better, easing any fears of manipulation of insufficient discrimination.  It seems like the Arimaa community keeps on growing and maturing.

Good luck to the defenders of humanity's honor in the coming weeks!


Arimaa Forum » Powered by YaBB 1 Gold - SP 1.3.1!
YaBB © 2000-2003. All Rights Reserved.