Arimaa Forum - Print Page


    
      
        Arimaa Forum
        (http://arimaa.com/arimaa/forum/cgi/YaBB.cgi)
      

        Arimaa >> Events >> Qualifying via highest rating
        
(Message started by: Fritzlein on Nov 24^th, 2004, 2:47pm)

Title: Qualifying via highest rating
Post by Fritzlein on Nov 24^th, 2004, 2:47pm

As I look at the recent Clueless vs. Speedy matches, it makes me think there is a potential problem with the system that has bots qualify for the computer championship on the basis of ratings.

Speedy has a much higher rating than Bomb from playing humans at a fast time control, since speeding up hurts programs less than it hurts humans. But presumably programs are roughly equally affected by playing quickly, so Clueless has roughly the same chance to beat Speedy at 30 seconds per move as to beat Bomb at 120 seconds per move. Thus by playing Speedy, Clueless can get an inflated rating.

This may not matter, since Clueless has a high enough rating to be in the top four with or without playing Speedy. Even so, it there happened to be five contenders for four spots, it wouldn't seem fair for the 30-second games to count towards Clueless's rating.

I would suggest a new rule: During November and December, no bots can vary their playing speed. If you want to play at two speeds, you have to get two accounts, like Bomb/Speedy. This prevents possible ratings distortion.

Title: Re: Qualifying via highest rating
Post by jdb on Nov 24^th, 2004, 3:03pm

Your point is valid.

Another alternative would be to compute a separate "qualifying rating", for the bots, based only on games that are admissible for qualification.

Title: Re: Qualifying via highest rating
Post by 99of9 on Nov 25^th, 2004, 3:09pm

on 11/24/04 at 14:47:44, Fritzlein wrote:

Speedy has a much higher rating than Bomb from playing humans at a fast time control, since speeding up hurts programs less than it hurts humans.

This is precisely the reason my rating is so high too! You guys are just much easier to beat at fast time control ;-).

Title: Re: Qualifying via highest rating
Post by omar on Nov 27^th, 2004, 3:47am

We could still have a problem if bot_clueless was waiting for a game and David decides to run bot_Speedy against it. bot_clueless would not be varying its playing speed in this case but still get in games against bot_Speedy.

So in addition to what Fritzlein proposed, we may also need to only count games that are played against other bots that are trying to qualify (as jdb proposed).

Another possibilty may be to use a ladder system among the bots that are qualifying and select the top 4 bots from the ladder rather than from the ratings.

Title: Re: Qualifying via highest rating
Post by fotland on Nov 27^th, 2004, 12:20pm

How about not limiting the number of bot entries to four? Since they are run automatically, it doesn't take much more effort to include more bots. You just need to separate the serious efforts from frivolous. Perhaps just pick a bot as a qualifier, and say that the bot needs to play enough games, and have a rating above 1600, or beat the reference bot, say 6 or more games out of 10.

Title: Re: Qualifying via highest rating
Post by 99of9 on Nov 27^th, 2004, 12:28pm

I think David's ideas are excellent. I would like to see the round robin including all interested bot developers - even if the bots aren't up to scratch. Last year I entered Gnobot for this very reason, even though I knew it was terrible compared to bomb / occam.

Win 5 out of 10 against a (randomized) arimaazilla-level bot, and you're in as long as you play the requisite number of games during December (so everyone can see your bot play).

Title: Re: Qualifying via highest rating
Post by omar on Dec 5^th, 2004, 6:08pm

Well more games does add to the cost, since I would need to rent the servers longer and pay more to the ICGA to conduct the tournament. I don't think we will have any problems this year. The top 4 this year seem to be:

bot_Bomb
bot_GnoBot
bot_clueless
bot_Loc

I don't think bot_Occam, bot_Aamira or the other bots are trying to qualify this year.

If there are more bots next year, maybe we can have a tounament with all the bots playing to narrow down the field to 4. The bots would be running on the developer's system and under the developers control during this tournament. So we can use the result of the tournament instead of rating to pick the top 4 bots.

I would still require that the bots be made availible online to play some games against humans and other bots because I don't want to have a situations like what happened with Deep Blue.