Arimaa Forum - Print Page


    
      
        Arimaa Forum
        (http://arimaa.com/arimaa/forum/cgi/YaBB.cgi)
      

        Arimaa >> Events >> Computer Championship format for 2006
        
(Message started by: omar on Aug 15^th, 2005, 4:27pm)

Title: Computer Championship format for 2006
Post by omar on Aug 15^th, 2005, 4:27pm

http://arimaa.com/arimaa/wcc/2006/

With computer we can try a floating triple elimination format.

I had run simulations to compare a double round robin against FTE for four players and suprisingly FTE did better than DRR even with high inaccuracy levels.

FTE with inacc=400: 65.4%
DRR with inacc=400: 56.4%

Another nice thing about using FTE is that the tournament now does not limit the number of entries to just four.

Title: Re: Computer Championship format for 2006
Post by 99of9 on Aug 15^th, 2005, 6:01pm

Great.

Title: Re: Computer Championship format for 2006
Post by Fritzlein on Aug 15^th, 2005, 6:27pm

on 08/15/05 at 16:27:02, omar wrote:

Another nice thing about using FTE is that the tournament now does not limit the number of entries to just four.

I'm thrilled that you will be allowing up to eight bots this year!

Title: Re: Computer Championship format for 2006
Post by Fritzlein on Aug 15^th, 2005, 11:05pm

The current qualifying hurdle is to play at least 4 different opponents for a total of 10 games, including at least 2 human opponents for a total of 4 games. This seems a bit on the low side to me in terms of human play. The bot vs. bot games are cheap, since they can perfectly well all come against Arimaalon and Arimaazilla. What really gives information is the bot vs. human games.

What would people say to 12 games including at least 3 human opponents for at least 6 games? There are significantly more humans around than there used to be, so that shouldn't be too onerous, should it? What was the experience of bot developers last year? Was it difficult to find enough human opponents over a two-month span?

Title: Re: Computer Championship format for 2006
Post by PMertens on Aug 16^th, 2005, 12:01am

I doubt the hurdle is a problem ... do you expect more than 8 bots to compete ?

Title: Re: Computer Championship format for 2006
Post by omar on Aug 17^th, 2005, 5:56pm

on 08/15/05 at 23:05:55, Fritzlein wrote:

True, but look at it from the bot developers point of view. A developer might not want too much to be known about the bot. Remember how deep blue never played any official games until playing GK. I definitely don't want to allow total secrecy either. I think something in between is a good compromise.

Quote:

What would people say to 12 games including at least 3 human opponents for at least 6 games? There are significantly more humans around than there used to be, so that shouldn't be too onerous, should it? What was the experience of bot developers last year? Was it difficult to find enough human opponents over a two-month span?

Last year it was 20 rated games against at least 6 different opponents using the 2/2/100/10/8 time control. At least 6 of those games must be played against at least 3 different human opponents

Maybe I should just leave that part the same.

Title: Re: Computer Championship format for 2006
Post by omar on Aug 27^th, 2005, 11:00am

I think it might be better to require the bots to be made available to play for a certian number of days shortly before the WCC tournament rather than requiring a certian number of games. A bot developer has more direct control over this.

For example requiring the qualifying bots to be online for 40 days during November and December for anyone to play against.

Title: Re: Computer Championship format for 2006
Post by Fritzlein on Aug 27^th, 2005, 12:02pm

on 08/27/05 at 11:00:55, omar wrote:

For example requiring the qualifying bots to be online for 40 days during November and December for anyone to play against.

I like this idea of time online rather than number of games, but does every bot developer have an always-on Internet connection? Maybe you could host bots for developers for whom this requirement would be a hardship.

I also like the increase in availability this represents. Even logging 20 full days (480 total hours) would create more opportunities than the way you had the rules.

Title: Re: Computer Championship format for 2006
Post by Fritzlein on Aug 27^th, 2005, 12:09pm

on 08/17/05 at 17:56:07, omar wrote:

True, but look at it from the bot developers point of view. A developer might not want too much to be known about the bot.

It's funny how different our intuitions are about the challenge format. From my perspective it seems reasonable to force developers to make their bots availble for quite a bit of play prior to the challenge match, but it seems undesirable to require a sweep of all three human opponents. You, on the other hand, feel comfortable requiring a sweep, but don't want to entirely deprive developers of the mystery factor.

I guess there are different ways to prevent the challenge from being won "by accident", and it comes down to a matter of personal taste which seems most appropriate.

I'm not trying to restart the discussion: I just think it is funny that in another thread I was taking the side of the developers in the threshold for winning, and in this thread I don't give a hoot what the developers want. I side with the human defenders in wanting more practice games, maybe because I'm hoping to be a human defender :D

Title: Re: Computer Championship format for 2006
Post by jdb on Aug 27^th, 2005, 12:19pm

Quote:

For example requiring the qualifying bots to be online for 40 days during November and December for anyone to play against.

Lasy year, I did not have an always on Internet connection. Leaving the bot online for extended periods of time on a dial up account is not practical.

If you want more games, consider cutting the time control to the POTM speed. I think people would be alot more willing to play at 45sec per move, than 120 sec. The bot's difference in playing strength would not be too great.

Online testing is a great way to test a bot. Developing a bot in secret would be alot more work because of the limited access to opponents.

Title: Re: Computer Championship format for 2006
Post by omar on Aug 27^th, 2005, 2:30pm

on 08/27/05 at 12:02:42, Fritzlein wrote:

I like this idea of time onlive rather than number of games, but does every bot developer have an always-on Internet connection? Maybe you could host bots for developers for whom this requirement would be a hardship.

Yes, in fact I can give an account on a Linux server that's connected to the Internet for any developer that needs it for developing and testing their bot.

Title: Re: Computer Championship format for 2006
Post by Fritzlein on Aug 27^th, 2005, 6:08pm

A minor point:

The tournament rules page says "Within each pairing, whoever has played Gold fewer times so far in the tournament should get to play Gold for that game, with ties broken randomly."

Now suppose Player A has played gold once and silver twice, whereas Player B has played gold once and silver once and has had a bye. It seems that Player A should get gold automatically, not with 50% probability. The two have each played gold the same number of times, but they've played silver a different number of times.

It's a tiny point, but a tiny fix will correct it: "Within each pairing, whoever has played Gold a lesser percentage of games so far in the tournament should get to play Gold for that game, with ties broken randomly."

Title: Re: Computer Championship format for 2006
Post by Fritzlein on Nov 9^th, 2005, 2:37pm

on 08/27/05 at 11:00:55, omar wrote:

I'm getting worried about this requirement. So far only Gnobot and Bomb are sitting on-line racking up time. I see doublep and JDB around, so I guess Aami-ra and Clueless will come online soon, but what about bot_haizhi? Omar, are you in touch with Haizhi? Do you know whether he plans to compete in the computer championship? It seems important to remind potential competitors about the qualifying hurdles in order to help them participate.

Also, I'm worried about the "playing at full strength" clause, because Bomb is obviously playing much worse than usual. Are these days of Bomb being online not counting towards the total?

Title: Re: Computer Championship format for 2006
Post by 99of9 on Nov 9^th, 2005, 8:20pm

on 11/09/05 at 14:37:44, Fritzlein wrote:

I'm getting worried about this requirement. So far only Gnobot and Bomb are sitting on-line racking up time.

Since Gnobot runs on a computer which has a different primary purpose, it is only available on the weekend and during the Australian nighttime. Therefore I suggest those who want to play it make use of whatever hours they do overlap.

Please let me know if you try to play Gnobot but it somehow fails - I'm getting occasional errors from the bot interface program.

Quote:

Are these days of Bomb being online not counting towards the total?

If David had deliberately made the online version substandard, I think that would be true. But I think we need to trust that it was just an error and ask that he corrects it soon and continues to put bomb online regularly. If he does that I think this time should count for bomb.

Title: Re: Computer Championship format for 2006
Post by omar on Nov 15^th, 2005, 4:54pm

David mentioned in the comments of game 21447 that he forgot to add the registration code after the hard drive crash, so I think it was definitely unintentional. I agree with Toby that in this case the previous time online should count.

Title: Re: Computer Championship format for 2006
Post by 99of9 on Nov 15^th, 2005, 6:16pm

Unfortunately even since he made that comment, someone mentioned that bomb has occasionally been playing a little strangely.

Title: Re: Computer Championship format for 2006
Post by BlackKnight on Nov 16^th, 2005, 7:02am

That is true. Even always strangely playing Loc won today against Bomb that finally even lost on time.

Title: Re: Computer Championship format for 2006
Post by omar on Nov 16^th, 2005, 5:26pm

I wonder how it will do against Bomb2005CC? I'll run a few games against it.

BTW I don't know if the bot developers know this, but you can start up the last years bots and have your current bot play against it. For example after starting up the last years bot, just type something like:
bot play bot_Bomb2005CC
at the command promt to have your bot play against it.

I just wanted to mention it in case someone didn't know.

Title: Re: Computer Championship format for 2006
Post by 99of9 on Nov 16^th, 2005, 9:43pm

We now only have a few days left for bots to announce their availability. Has anyone emailed Haizhi yet? It'd be really great for his bot to take part - especially since there was an entire thesis written about it.

Title: Re: Computer Championship format for 2006
Post by BlackKnight on Nov 17^th, 2005, 1:55am

on 11/16/05 at 17:26:48, omar wrote:

BTW I don't know if the bot developers know this, but you can start up the last years bots and have your current bot play against it. For example after starting up the last years bot, just type something like:
bot play bot_Bomb2005CC
at the command promt to have your bot play against it.

Thanks a lot for that reminder.

Do you think it is difficult to extend the script in such a way so that "bot play bot_XXX2005CC" also wakes up that bot.

Title: Re: Computer Championship format for 2006
Post by omar on Nov 19^th, 2005, 10:03am

on 11/17/05 at 01:55:42, BlackKnight wrote:

Do you think it is difficult to extend the script in such a way so that "bot play bot_XXX2005CC" also wakes up that bot.

That should probably be done in a higher leve script. For example:

matchOnline bot_bomb 5

would start up bot_bomb and then start your bot to play against it; and would repeat this so that 5 games are played.

The matchOnline script would need to use something like wget to started up the specified bot.

Title: Re: Computer Championship format for 2006
Post by nbarriga on Nov 19^th, 2005, 11:04am

What about this?

#!/bin/sh
for i in $(seq 1 $3);do
wget http://arimaa.com/arimaa/bots/$1/controlBot.cgi --post-data="side=$2&start=Start%20Bot" ;
sleep 3;
./bot play $1 &
done

Just save it in the directory where the "bot" script is as matchOnline. To use:
./matchOnline bot_Loc2005P1 w 3

This will play your bot 3 times, with Loc2005P1 playing gold.

Title: Re: Computer Championship format for 2006
Post by nbarriga on Nov 19^th, 2005, 11:42am

on 11/19/05 at 11:04:51, nbarriga wrote:

#!/bin/sh
for i in $(seq 1 $3);do
wget http://arimaa.com/arimaa/bots/$1/controlBot.cgi --post-data="side=$2&start=Start%20Bot" ;
sleep 3;
./bot play $1 &
done

Just 2 fixes:
./bot play $1 (without &)
and add rm -f controlBot.cgi

So the final version:
#!/bin/sh
for i in $(seq 1 $3);do
wget http://arimaa.com/arimaa/bots/$1/controlBot.cgi --post-data="side=$2&start=Start%20Bot" ;
sleep 3;
rm -f controlBot.cgi
./bot play $1
done

Title: Re: Computer Championship format for 2006
Post by BlackKnight on Nov 19^th, 2005, 11:00pm

Thanks a lot to both of you! :-D

Title: Re: Computer Championship format for 2006
Post by omar on Dec 20^th, 2005, 12:45pm

I've decided to postpone the start of the computer championship tournament by one week, since I will be out of the country between Jan 1st and Jan 15th and will not have regular internet access. I don't want to risk a technical problem occuring while Im unreachable. Three servers have to be running smoothly for the computer championship to run smoothly. So I think it's better to postpone it till Jan 16th. I think the tournament should still be able to finish before the end of January.

Title: Re: Computer Championship format for 2006
Post by 99of9 on Dec 20^th, 2005, 2:26pm

Sounds good. I don't think the computers will mind a packed schedule :-).

Title: Re: Computer Championship format for 2006
Post by Fritzlein on Dec 22^nd, 2005, 3:58pm

When will we know the list of participants? Is it finalized on December 31?

Title: Re: Computer Championship format for 2006
Post by omar on Dec 26^th, 2005, 8:58pm

Looks like the bots this year are going to be:

Bomb, GnoBot, Clueless, Loc and Aamira

Haizhi had wanted to particpate also, but he said he recently got a new job and won't have the time to. I offered to waive the qualifying games and help him get the bot setup, but have not heard back yet. I would really like to see his bot play in the tournament also.

Title: Re: Computer Championship format for 2006
Post by omar on Dec 29^th, 2005, 5:14am

Im going to have to change the start of the computer championship to Jan 18th, because I don't think I will be having regular internet access till then. Im still planning to finish the tournament before the end of January.