Author |
Topic: Will the 2010 Computer Championship be open? (Read 13768 times) |
|
omar
Forum Guru
Arimaa player #2
Gender: 
Posts: 1003
|
 |
Re: Will the 2010 Computer Championship be open?
« Reply #135 on: Jan 22nd, 2010, 8:16am » |
 Quote  Modify
|
I checked the options for GnoBot 2009 and I didn't see anything for limiting the number of CPUs that it uses. Also it does a lot of disk IO and that also slows down other processes. I the TD approves I will remove bot_GnoBot2009Blitz from the list of benchmark bots.
|
| « Last Edit: Jan 22nd, 2010, 8:19am by omar » |
 IP Logged |
|
|
|
RonWeasley
Forum Guru
Harry's friend (Arimaa player #441)
Gender:
Posts: 882
|
|
Re: Will the 2010 Computer Championship be open?
« Reply #136 on: Jan 25th, 2010, 10:16am » |
Quote Modify
|
OK. This seems like the best consensus we're going to achieve. Go ahead and remove bot_GnoBot2009Blitz from the benchmark list.
|
|
IP Logged |
|
|
|
jdb
Forum Guru
Arimaa player #214
Gender:
Posts: 682
|
|
Re: Will the 2010 Computer Championship be open?
« Reply #137 on: Jan 29th, 2010, 5:30pm » |
Quote Modify
|
on Jan 25th, 2010, 10:16am, RonWeasley wrote:| OK. This seems like the best consensus we're going to achieve. Go ahead and remove bot_GnoBot2009Blitz from the benchmark list. |
|
Would it be possible to do this before the qualifying period ends? It can be figured out manually, so its not overly critical.
|
|
IP Logged |
|
|
|
99of9
Forum Guru
Gnobby's creator (player #314)
 
Gender:
Posts: 1413
|
|
Re: Will the 2010 Computer Championship be open?
« Reply #138 on: Jan 30th, 2010, 2:18am » |
Quote Modify
|
And omar, if you have any other spare time this weekend, please can you run lots more qualifying games for bomb? Correct seeding may require up to another 53 games!
|
|
IP Logged |
|
|
|
Tuks
Forum Guru
Arimaa player #2626
Gender:
Posts: 203
|
|
Re: Will the 2010 Computer Championship be open?
« Reply #139 on: Jan 30th, 2010, 3:15am » |
Quote Modify
|
and the others need to start playing, the whole point of the qualifiers is to rank the bots, if you dont play all your games the rankings are completely inaccurate and end up being pointless or even unfair.
|
|
IP Logged |
|
|
|
doublep
Forum Guru
Badger author
Gender:
Posts: 82
|
|
Re: Will the 2010 Computer Championship be open?
« Reply #140 on: Jan 30th, 2010, 7:16am » |
Quote Modify
|
Maybe qualifying period could be extended another week or so? E.g. OpFor has not played top bots at all and risks receiving 8th seed whereas it would be 4th or 5th otherwise at least.
|
|
IP Logged |
|
|
|
doublep
Forum Guru
Badger author
Gender:
Posts: 82
|
|
Re: Will the 2010 Computer Championship be open?
« Reply #141 on: Jan 30th, 2010, 7:24am » |
Quote Modify
|
Also, I think if think if this qualifying scheme will be preserved, we should cut down on the weaker opponent. E.g. of the 4 weaker bots leave at most two (Arimaazilla and Aami-ra). Or maybe remove them all; contestants are expected to become stronger each next year, so weaker qualifying opponents will make even less of a difference.
And please reduce max. number of games per opponent from 20 to something like 10. Otherwise current situation with too many unplayed games which could possibly improve seeding, will certainly repeat.
|
|
IP Logged |
|
|
|
Fritzlein
Forum Guru
Arimaa player #706
Gender:
Posts: 5928
|
|
Re: Will the 2010 Computer Championship be open?
« Reply #142 on: Jan 30th, 2010, 7:39am » |
Quote Modify
|
Agreed. The weaker bots were too weak, and 20 games per bot is too many. I supported the "winning streak" idea at first, but I have since totally changed my mind.
Next year I propose that the stable consist of the blitz version of eight different bots (no P1, P2, or Fast), and the qualifying be exactly four games against each, i.e. two as Gold and two as Silver. Add up the wins, and that's the qualifying score.
The idea behind having such a huge number of potential games was to permit developers to make last-minute modifications to their bots after losing to a benchmark bot. Mostly these last-minute modifications have turned out to be fictitious, and to the extent that they have been genuine, it has been a bad thing for the qualifying because we now have one day left and huge numbers of unplayed games. Either those games will never be played, distorting the standings, or they will all be played at the last minute, potentially overloading the server and disrupting the human World Championship.
Lets get back to something manageable with 8x4=32 games per qualifying bot. That's pretty close to what developers have decided to play anyway.
|
|
IP Logged |
|
|
|
doublep
Forum Guru
Badger author
Gender:
Posts: 82
|
|
Re: Will the 2010 Computer Championship be open?
« Reply #143 on: Jan 30th, 2010, 8:06am » |
Quote Modify
|
Well, at least for Badger those last-minute changes were important. E.g. it now has a fair chance again OpFor2009 whereas it lost 7 games in row before. But I agree they don't justify sheer number of qualifying games. If format was different with fewer games, I could just train and debug against other bots.
Having only blitz is perhaps too severe. After all, some bots can be natural "slow thinkers" and such qualifying scheme would unfairly disadvantage them. I'd propose to have 4 to 6 blitz bots and 2 to 4 fast opponents instead.
|
|
IP Logged |
|
|
|
jdb
Forum Guru
Arimaa player #214
Gender:
Posts: 682
|
|
Re: Will the 2010 Computer Championship be open?
« Reply #144 on: Jan 30th, 2010, 8:14am » |
Quote Modify
|
For me, the problem was having to manually start each game. The games against the fast bots take around an hour, so all I can do is get in a few games a day. I have to be at the computer to start the next game. If there were a script to automatically play the games it would make it much easier.
|
|
IP Logged |
|
|
|
Fritzlein
Forum Guru
Arimaa player #706
Gender:
Posts: 5928
|
|
Re: Will the 2010 Computer Championship be open?
« Reply #145 on: Jan 30th, 2010, 8:54am » |
Quote Modify
|
on Jan 30th, 2010, 8:06am, doublep wrote:| Having only blitz is perhaps too severe. After all, some bots can be natural "slow thinkers" and such qualifying scheme would unfairly disadvantage them. I'd propose to have 4 to 6 blitz bots and 2 to 4 fast opponents instead. |
|
As I recall, Gnobot2009 did better against Bomb2005 at blitz than at slower speeds, so it is true that some bots are natural slow/fast thinkers.
However, by proposing only blitz games I am thinking of the time commitment (see jdb's post) and server resources. Each blitz game uses (on average) half the server CPU of a fast game, because it only lasts half as long. Therefore the total server CPU consumption is minimized by having all qualifying games played at blitz speed. I consider this a weightier consideration than accommodating naturally slow-thinking bots.
|
|
IP Logged |
|
|
|
Fritzlein
Forum Guru
Arimaa player #706
Gender:
Posts: 5928
|
|
Re: Will the 2010 Computer Championship be open?
« Reply #146 on: Jan 30th, 2010, 9:08am » |
Quote Modify
|
on Jan 22nd, 2010, 8:16am, omar wrote:| I checked the options for GnoBot 2009 and I didn't see anything for limiting the number of CPUs that it uses. Also it does a lot of disk IO and that also slows down other processes. I the TD approves I will remove bot_GnoBot2009Blitz from the list of benchmark bots. |
|
Thanks for fixing the standings, Omar. That makes it much easier to see that marwin has the #1 seed locked up, and that both GnoBot and clueless have maxed out all the bots except Clueless2009Blitz, so that JDB probably doesn't need to play any more games to have the #2 seed unless GnoBot unexpectedly pulls out a four-game winning streak against Clueless2009Blitz, etc.
|
| « Last Edit: Jan 30th, 2010, 9:13am by Fritzlein » |
IP Logged |
|
|
|
omar
Forum Guru
Arimaa player #2
Gender:
Posts: 1003
|
|
Re: Will the 2010 Computer Championship be open?
« Reply #147 on: Jan 30th, 2010, 9:08am » |
Quote Modify
|
I've removed GnoBot2009Blitz from the list of benchmark bots.
Toby, I don't think I'll be able to run too many more games. I'll try to at least have Bomb play at least 4 games against all the benchmark bots.
|
|
IP Logged |
|
|
|
aaaa
Forum Guru
Arimaa player #958
Posts: 768
|
|
Re: Will the 2010 Computer Championship be open?
« Reply #148 on: Jan 30th, 2010, 9:13am » |
Quote Modify
|
I say we just scratch the whole idea of benchmark bots and next time put no limit to the number of bots that can participate in the championship. Any seeding could possibly be determined by a (thus voluntary) tournament involving developer bots, perhaps in a style similar to that of the Continuous Tournament.
In order to make the championship fit in the allotted time frame, automatize the whole process of scheduling games (maybe allowing for some pre- and post-game processing). Switching to a round-robin/elimination hybrid tournament format would then also follow naturally.
Let's not be held back by any supposed consideration towards spectators. So what, if one won't be able anymore to follow live as large a proportion of games as before, which are already pretty unfriendly towards spectators being 2 minutes per move? Why should that be allowed to stand in the way of any thorough way of determining what is the best bot?
I know this all has been said before, but surely a year should be enough time to work things out, right?
|
|
IP Logged |
|
|
|
Fritzlein
Forum Guru
Arimaa player #706
Gender:
Posts: 5928
|
|
Re: Will the 2010 Computer Championship be open?
« Reply #149 on: Jan 30th, 2010, 9:40am » |
Quote Modify
|
on Jan 30th, 2010, 9:13am, aaaa wrote:| I say we just scratch the whole idea of benchmark bots and next time put no limit to the number of bots that can participate in the championship. |
|
As Omar pointed out earlier, if we use the finite resources more efficiently, the number of participating bots could be greater, but not unlimited. A qualifying procedure will eventually have to be implemented regardless.
That said, I do like the idea of having more bots in the main tournament, even if the games are automated to run 24/7. That's not spectator-unfriendly: on the contrary it means that there is always a tournament game to watch. It would be a spectator's dream!
|
|
IP Logged |
|
|
|
|
Home
Help
Search
Members
Login
Register
Reply
Notify of replies
Send Topic
Print