Author |
Topic: Qualifying via highest rating (Read 951 times) |
|
Fritzlein
Forum Guru
Arimaa player #706
Gender: 
Posts: 5928
|
 |
Qualifying via highest rating
« on: Nov 24th, 2004, 2:47pm » |
 Quote  Modify
|
As I look at the recent Clueless vs. Speedy matches, it makes me think there is a potential problem with the system that has bots qualify for the computer championship on the basis of ratings.
Speedy has a much higher rating than Bomb from playing humans at a fast time control, since speeding up hurts programs less than it hurts humans. But presumably programs are roughly equally affected by playing quickly, so Clueless has roughly the same chance to beat Speedy at 30 seconds per move as to beat Bomb at 120 seconds per move. Thus by playing Speedy, Clueless can get an inflated rating.
This may not matter, since Clueless has a high enough rating to be in the top four with or without playing Speedy. Even so, it there happened to be five contenders for four spots, it wouldn't seem fair for the 30-second games to count towards Clueless's rating.
I would suggest a new rule: During November and December, no bots can vary their playing speed. If you want to play at two speeds, you have to get two accounts, like Bomb/Speedy. This prevents possible ratings distortion.
|
|
 IP Logged |
|
|
|
jdb
Forum Guru
Arimaa player #214
Gender:
Posts: 682
|
|
Re: Qualifying via highest rating
« Reply #1 on: Nov 24th, 2004, 3:03pm » |
Quote Modify
|
Your point is valid.
Another alternative would be to compute a separate "qualifying rating", for the bots, based only on games that are admissible for qualification.
|
|
IP Logged |
|
|
|
99of9
Forum Guru
Gnobby's creator (player #314)
 
Gender:
Posts: 1413
|
|
Re: Qualifying via highest rating
« Reply #2 on: Nov 25th, 2004, 3:09pm » |
Quote Modify
|
on Nov 24th, 2004, 2:47pm, Fritzlein wrote:
Speedy has a much higher rating than Bomb from playing humans at a fast time control, since speeding up hurts programs less than it hurts humans. |
|
This is precisely the reason my rating is so high too! You guys are just much easier to beat at fast time control  .
|
|
IP Logged |
|
|
|
omar
Forum Guru
Arimaa player #2
Gender:
Posts: 1003
|
|
Re: Qualifying via highest rating
« Reply #3 on: Nov 27th, 2004, 3:47am » |
Quote Modify
|
We could still have a problem if bot_clueless was waiting for a game and David decides to run bot_Speedy against it. bot_clueless would not be varying its playing speed in this case but still get in games against bot_Speedy.
So in addition to what Fritzlein proposed, we may also need to only count games that are played against other bots that are trying to qualify (as jdb proposed).
Another possibilty may be to use a ladder system among the bots that are qualifying and select the top 4 bots from the ladder rather than from the ratings.
|
|
IP Logged |
|
|
|
fotland
Forum Guru
Arimaa player #211
Gender:
Posts: 216
|
|
Re: Qualifying via highest rating
« Reply #4 on: Nov 27th, 2004, 12:20pm » |
Quote Modify
|
How about not limiting the number of bot entries to four? Since they are run automatically, it doesn't take much more effort to include more bots. You just need to separate the serious efforts from frivolous. Perhaps just pick a bot as a qualifier, and say that the bot needs to play enough games, and have a rating above 1600, or beat the reference bot, say 6 or more games out of 10.
|
|
IP Logged |
|
|
|
99of9
Forum Guru
Gnobby's creator (player #314)
Gender:
Posts: 1413
|
|
Re: Qualifying via highest rating
« Reply #5 on: Nov 27th, 2004, 12:28pm » |
Quote Modify
|
I think David's ideas are excellent. I would like to see the round robin including all interested bot developers - even if the bots aren't up to scratch. Last year I entered Gnobot for this very reason, even though I knew it was terrible compared to bomb / occam.
Win 5 out of 10 against a (randomized) arimaazilla-level bot, and you're in as long as you play the requisite number of games during December (so everyone can see your bot play).
|
|
IP Logged |
|
|
|
omar
Forum Guru
Arimaa player #2
Gender:
Posts: 1003
|
|
Re: Qualifying via highest rating
« Reply #6 on: Dec 5th, 2004, 6:08pm » |
Quote Modify
|
Well more games does add to the cost, since I would need to rent the servers longer and pay more to the ICGA to conduct the tournament. I don't think we will have any problems this year. The top 4 this year seem to be:
bot_Bomb
bot_GnoBot
bot_clueless
bot_Loc
I don't think bot_Occam, bot_Aamira or the other bots are trying to qualify this year.
If there are more bots next year, maybe we can have a tounament with all the bots playing to narrow down the field to 4. The bots would be running on the developer's system and under the developers control during this tournament. So we can use the result of the tournament instead of rating to pick the top 4 bots.
I would still require that the bots be made availible online to play some games against humans and other bots because I don't want to have a situations like what happened with Deep Blue.
|
|
IP Logged |
|
|
|
|
Home
Help
Search
Members
Login
Register
Reply
Notify of replies
Send Topic
Print