Author |
Topic: Computer Championship format for 2006 (Read 2507 times) |
|
omar
Forum Guru
Arimaa player #2
Gender: 
Posts: 1003
|
 |
Computer Championship format for 2006
« on: Aug 15th, 2005, 4:27pm » |
 Quote  Modify
|
http://arimaa.com/arimaa/wcc/2006/
With computer we can try a floating triple elimination format.
I had run simulations to compare a double round robin against FTE for four players and suprisingly FTE did better than DRR even with high inaccuracy levels.
FTE with inacc=400: 65.4%
DRR with inacc=400: 56.4%
Another nice thing about using FTE is that the tournament now does not limit the number of entries to just four.
|
|
 IP Logged |
|
|
|
99of9
Forum Guru
Gnobby's creator (player #314)
 
Gender:
Posts: 1413
|
|
Re: Computer Championship format for 2006
« Reply #1 on: Aug 15th, 2005, 6:01pm » |
Quote Modify
|
Great.
|
|
IP Logged |
|
|
|
Fritzlein
Forum Guru
Arimaa player #706
Gender:
Posts: 5928
|
|
Re: Computer Championship format for 2006
« Reply #2 on: Aug 15th, 2005, 6:27pm » |
Quote Modify
|
on Aug 15th, 2005, 4:27pm, omar wrote:Another nice thing about using FTE is that the tournament now does not limit the number of entries to just four.
|
|
I'm thrilled that you will be allowing up to eight bots this year!
|
|
IP Logged |
|
|
|
Fritzlein
Forum Guru
Arimaa player #706
Gender:
Posts: 5928
|
|
Re: Computer Championship format for 2006
« Reply #3 on: Aug 15th, 2005, 11:05pm » |
Quote Modify
|
The current qualifying hurdle is to play at least 4 different opponents for a total of 10 games, including at least 2 human opponents for a total of 4 games. This seems a bit on the low side to me in terms of human play. The bot vs. bot games are cheap, since they can perfectly well all come against Arimaalon and Arimaazilla. What really gives information is the bot vs. human games.
What would people say to 12 games including at least 3 human opponents for at least 6 games? There are significantly more humans around than there used to be, so that shouldn't be too onerous, should it? What was the experience of bot developers last year? Was it difficult to find enough human opponents over a two-month span?
|
| « Last Edit: Aug 15th, 2005, 11:09pm by Fritzlein » |
IP Logged |
|
|
|
PMertens
Forum Guru
Arimaa player #692
Gender:
Posts: 437
|
|
Re: Computer Championship format for 2006
« Reply #4 on: Aug 16th, 2005, 12:01am » |
Quote Modify
|
I doubt the hurdle is a problem ... do you expect more than 8 bots to compete ?
|
|
IP Logged |
|
|
|
omar
Forum Guru
Arimaa player #2
Gender:
Posts: 1003
|
|
Re: Computer Championship format for 2006
« Reply #5 on: Aug 17th, 2005, 5:56pm » |
Quote Modify
|
on Aug 15th, 2005, 11:05pm, Fritzlein wrote:The current qualifying hurdle is to play at least 4 different opponents for a total of 10 games, including at least 2 human opponents for a total of 4 games. This seems a bit on the low side to me in terms of human play. The bot vs. bot games are cheap, since they can perfectly well all come against Arimaalon and Arimaazilla. What really gives information is the bot vs. human games.
|
|
True, but look at it from the bot developers point of view. A developer might not want too much to be known about the bot. Remember how deep blue never played any official games until playing GK. I definitely don't want to allow total secrecy either. I think something in between is a good compromise.
Quote:
What would people say to 12 games including at least 3 human opponents for at least 6 games? There are significantly more humans around than there used to be, so that shouldn't be too onerous, should it? What was the experience of bot developers last year? Was it difficult to find enough human opponents over a two-month span?
|
|
Last year it was 20 rated games against at least 6 different opponents using the 2/2/100/10/8 time control. At least 6 of those games must be played against at least 3 different human opponents
Maybe I should just leave that part the same.
|
|
IP Logged |
|
|
|
omar
Forum Guru
Arimaa player #2
Gender:
Posts: 1003
|
|
Re: Computer Championship format for 2006
« Reply #6 on: Aug 27th, 2005, 11:00am » |
Quote Modify
|
I think it might be better to require the bots to be made available to play for a certian number of days shortly before the WCC tournament rather than requiring a certian number of games. A bot developer has more direct control over this.
For example requiring the qualifying bots to be online for 40 days during November and December for anyone to play against.
|
|
IP Logged |
|
|
|
Fritzlein
Forum Guru
Arimaa player #706
Gender:
Posts: 5928
|
|
Re: Computer Championship format for 2006
« Reply #7 on: Aug 27th, 2005, 12:02pm » |
Quote Modify
|
on Aug 27th, 2005, 11:00am, omar wrote:For example requiring the qualifying bots to be online for 40 days during November and December for anyone to play against.
|
|
I like this idea of time online rather than number of games, but does every bot developer have an always-on Internet connection? Maybe you could host bots for developers for whom this requirement would be a hardship.
I also like the increase in availability this represents. Even logging 20 full days (480 total hours) would create more opportunities than the way you had the rules.
|
| « Last Edit: Sep 10th, 2005, 4:43am by Fritzlein » |
IP Logged |
|
|
|
Fritzlein
Forum Guru
Arimaa player #706
Gender:
Posts: 5928
|
|
Re: Computer Championship format for 2006
« Reply #8 on: Aug 27th, 2005, 12:09pm » |
Quote Modify
|
on Aug 17th, 2005, 5:56pm, omar wrote:| True, but look at it from the bot developers point of view. A developer might not want too much to be known about the bot. |
|
It's funny how different our intuitions are about the challenge format. From my perspective it seems reasonable to force developers to make their bots availble for quite a bit of play prior to the challenge match, but it seems undesirable to require a sweep of all three human opponents. You, on the other hand, feel comfortable requiring a sweep, but don't want to entirely deprive developers of the mystery factor.
I guess there are different ways to prevent the challenge from being won "by accident", and it comes down to a matter of personal taste which seems most appropriate.
I'm not trying to restart the discussion: I just think it is funny that in another thread I was taking the side of the developers in the threshold for winning, and in this thread I don't give a hoot what the developers want. I side with the human defenders in wanting more practice games, maybe because I'm hoping to be a human defender 
|
| « Last Edit: Aug 27th, 2005, 12:17pm by Fritzlein » |
IP Logged |
|
|
|
jdb
Forum Guru
Arimaa player #214
Gender:
Posts: 682
|
|
Re: Computer Championship format for 2006
« Reply #9 on: Aug 27th, 2005, 12:19pm » |
Quote Modify
|
Quote:For example requiring the qualifying bots to be online for 40 days during November and December for anyone to play against.
|
|
Lasy year, I did not have an always on Internet connection. Leaving the bot online for extended periods of time on a dial up account is not practical.
If you want more games, consider cutting the time control to the POTM speed. I think people would be alot more willing to play at 45sec per move, than 120 sec. The bot's difference in playing strength would not be too great.
Online testing is a great way to test a bot. Developing a bot in secret would be alot more work because of the limited access to opponents.
|
|
IP Logged |
|
|
|
omar
Forum Guru
Arimaa player #2
Gender:
Posts: 1003
|
|
Re: Computer Championship format for 2006
« Reply #10 on: Aug 27th, 2005, 2:30pm » |
Quote Modify
|
on Aug 27th, 2005, 12:02pm, Fritzlein wrote:
I like this idea of time onlive rather than number of games, but does every bot developer have an always-on Internet connection? Maybe you could host bots for developers for whom this requirement would be a hardship.
|
|
Yes, in fact I can give an account on a Linux server that's connected to the Internet for any developer that needs it for developing and testing their bot.
|
|
IP Logged |
|
|
|
Fritzlein
Forum Guru
Arimaa player #706
Gender:
Posts: 5928
|
|
Re: Computer Championship format for 2006
« Reply #11 on: Aug 27th, 2005, 6:08pm » |
Quote Modify
|
A minor point:
The tournament rules page says "Within each pairing, whoever has played Gold fewer times so far in the tournament should get to play Gold for that game, with ties broken randomly."
Now suppose Player A has played gold once and silver twice, whereas Player B has played gold once and silver once and has had a bye. It seems that Player A should get gold automatically, not with 50% probability. The two have each played gold the same number of times, but they've played silver a different number of times.
It's a tiny point, but a tiny fix will correct it: "Within each pairing, whoever has played Gold a lesser percentage of games so far in the tournament should get to play Gold for that game, with ties broken randomly."
|
| « Last Edit: Sep 2nd, 2005, 10:09pm by Fritzlein » |
IP Logged |
|
|
|
Fritzlein
Forum Guru
Arimaa player #706
Gender:
Posts: 5928
|
|
Re: Computer Championship format for 2006
« Reply #12 on: Nov 9th, 2005, 2:37pm » |
Quote Modify
|
on Aug 27th, 2005, 11:00am, omar wrote:I think it might be better to require the bots to be made available to play for a certian number of days shortly before the WCC tournament rather than requiring a certian number of games. A bot developer has more direct control over this.
For example requiring the qualifying bots to be online for 40 days during November and December for anyone to play against.
|
|
I'm getting worried about this requirement. So far only Gnobot and Bomb are sitting on-line racking up time. I see doublep and JDB around, so I guess Aami-ra and Clueless will come online soon, but what about bot_haizhi? Omar, are you in touch with Haizhi? Do you know whether he plans to compete in the computer championship? It seems important to remind potential competitors about the qualifying hurdles in order to help them participate.
Also, I'm worried about the "playing at full strength" clause, because Bomb is obviously playing much worse than usual. Are these days of Bomb being online not counting towards the total?
|
|
IP Logged |
|
|
|
99of9
Forum Guru
Gnobby's creator (player #314)
Gender:
Posts: 1413
|
|
Re: Computer Championship format for 2006
« Reply #13 on: Nov 9th, 2005, 8:20pm » |
Quote Modify
|
on Nov 9th, 2005, 2:37pm, Fritzlein wrote:| I'm getting worried about this requirement. So far only Gnobot and Bomb are sitting on-line racking up time. |
|
Since Gnobot runs on a computer which has a different primary purpose, it is only available on the weekend and during the Australian nighttime. Therefore I suggest those who want to play it make use of whatever hours they do overlap.
Please let me know if you try to play Gnobot but it somehow fails - I'm getting occasional errors from the bot interface program.
Quote:| Are these days of Bomb being online not counting towards the total? |
|
If David had deliberately made the online version substandard, I think that would be true. But I think we need to trust that it was just an error and ask that he corrects it soon and continues to put bomb online regularly. If he does that I think this time should count for bomb.
|
|
IP Logged |
|
|
|
omar
Forum Guru
Arimaa player #2
Gender:
Posts: 1003
|
|
Re: Computer Championship format for 2006
« Reply #14 on: Nov 15th, 2005, 4:54pm » |
Quote Modify
|
David mentioned in the comments of game 21447 that he forgot to add the registration code after the hard drive crash, so I think it was definitely unintentional. I agree with Toby that in this case the previous time online should count.
|
|
IP Logged |
|
|
|
|
Home
Help
Search
Members
Login
Register
Reply
Notify of replies
Send Topic
Print