Arimaa Forum - Print Page


    
      
        Arimaa Forum
        (http://arimaa.com/arimaa/forum/cgi/YaBB.cgi)
      

        Arimaa >> General Discussion >> ratings for 2010 CC bots
        
(Message started by: omar on Jul 29^th, 2010, 12:00am)

Title: ratings for 2010 CC bots
Post by omar on Jul 29^th, 2010, 12:00am

I just finished setting up the bots from the 2010 Computer Championship. Each bot has 5 versions:

CC - playing at time control of 2 minutes per move
P1 - search limited to 1 ply
P2 - search limited to 2 ply
Fast - playing at 30 seconds per move
Blitz - playing at 15 seconds per move

The ratings for these bots are currently set at 1400, but should be set something more reasonable based on ratings of the previous year bots or the developer bots ratings. Any suggestions on how to set the ratings?

Title: Re: ratings for 2010 CC bots
Post by rbarreira on Jul 29^th, 2010, 1:50am

Thanks a lot for the bots omar.

I was looking at the current ratings of bots from past years, here are my rough suggestions:

CC - Set them to the ratings at this page (http://arimaa.com/arimaa/wcc/2010/players.cgi). Maybe minus 100 points for the bots which are parallel if you want to. I think all of them are parallel except for bomb and sharp.

P1 - CC minus 300 points.

P2 - CC minus 150 points.

Fast - equal to the CC? not sure about this one.

Blitz - CC plus 100 points.

Title: Re: ratings for 2010 CC bots
Post by arimaa_master on Jul 29^th, 2010, 4:44am

on 07/29/10 at 00:00:33, omar wrote:

Thanks Omar, that is great.

However bot_Clueless2010P1 doesn´t seem to start properly.
Maybe there is some info missing at Control Page - Bot info: "not available" and Max bots allowed is: where should be number I guess.

Title: Re: ratings for 2010 CC bots
Post by rbarreira on Jul 29^th, 2010, 4:53am

Clueless2010P1 worked fine for me. Twice so far.

Maybe it only allows one instance of each bot?

Title: Re: ratings for 2010 CC bots
Post by Janzert on Jul 29^th, 2010, 5:18am

Hi Omar,

I'm guessing you haven't seen my recent post yet:

OpFor2010 fixed depth search bug (http://arimaa.com/arimaa/forum/cgi/YaBB.cgi?board=devTalk;action=display;num=1280184285)

Janzert

Title: Re: ratings for 2010 CC bots
Post by omar on Jul 29^th, 2010, 8:10am

rbarreira: thanks for the suggestion. Based on the time control most often played by the developer bot I guess the rating of the CC or Fast bot can be set. For example Marwin has earned its developer bot rating by playing mostly at CC speed whereas Clueless has played most of its games at the Fast speed. But how much to add or subtract to this base rating to get the ratings for the different versions is not clear. But then again, we just need to be in the ball park and not exact.

arimaa_master: I think I have all the bots set at only 1 instance right now. Will change the P1 and P2 bots to allow more instances.

Janzert: no, I had not seen that. I'll disable the P1 and P2 versions for now.

Title: Re: ratings for 2010 CC bots
Post by omar on Jul 29^th, 2010, 12:30pm

I should also mention that I've set all the bots to use only 1 core. I was going to let the CC bots use 4 cores, but I noticed this morning that the load went up very high and the server became very slow when onigawara was playing Marwin2010CC; so I've set all bots to use only 1 core. GnoBot does not provide an option to limit the number of cores it uses; so I've disabled CC, Fast and Blitz version (since even the Blitz version can think for minutes if it is in trouble and has enough reserve); the P1 and P2 versions are still available, but set to allow only 1 instance. The P1 and P2 versions of all other bots are set to allow up to 5 instances.

Title: Re: ratings for 2010 CC bots
Post by rbarreira on Jul 29^th, 2010, 12:49pm

I think it makes sense to set them to use one core, with 4 cores their performance would greatly vary according to the load on the server, which means their level of play would be quite unpredictable.

Of course the performance will still sometimes decrease even with just one core, but it's not that common for people to be playing four CC/Fast/Blitz bots simultaneously.

Title: Re: ratings for 2010 CC bots
Post by Fritzlein on Jul 29^th, 2010, 5:41pm

I agree with limiting bots to one core, and with disabling bots that don't give you the option to limit to one core. We should ask 99of9 about this; maybe there is a non-obvious flag in GnoBot 2010.

As for setting the ratings, I think you should just do it because people are already starting to play the bots. I suggest taking a base rating for a bot to be the rating of the developer bot. Then from the base rating.

bot_Blitz +100
bot_Fast +50
bot_CC -50
bot_P2 -100
bot_P1 -400

If the developer bot doesn't have a rating or you think it is inaccurate, key on one of last year's versions, with the relative gaps all the same. Obviously this won't work correctly, because different bots have different definitions of P1 and P2, depending on extensions. Still, as you say, we only need to get in the ballpark and then let the ratings float.

Title: Re: ratings for 2010 CC bots
Post by Fritzlein on Jul 29^th, 2010, 5:42pm

on 07/29/10 at 12:49:51, rbarreira wrote:

IOf course the performance will still sometimes decrease even with just one core, but it's not that common for people to be playing four CC/Fast/Blitz bots simultaneously.

Plus it is only the bot's move half of the time, so one should get approximately eight bots before the load starts to affect performance.

Title: Re: ratings for 2010 CC bots
Post by omar on Jul 30^th, 2010, 7:59am

Thanks for the input guys. I've reset the ratings for the 2010 bots. I basically used Karl's numbers except for P2 I used -150. Also I rounded the developer bot rating to the nearest 25; OpFor 1873 became 1875; Sharp 1958 became 1950, etc.

The bots can be started from this page:
http://arimaa.com/arimaa/bots/index.cgi

Title: Re: ratings for 2010 CC bots
Post by Tuks on Jul 30^th, 2010, 11:55am

could you add them to the bot ladder?

not all of them or even most of them, just a couple of the highest level ones like marwin and clueless, i understand that the bot ladder is to get people interested but its also like a training and 3 years ago a player at bombs level was top notch, now its the norm for someone to eventually get to bombs level,

i liked the fact that the completion of the bot ladder was an accomplishment

Title: Re: ratings for 2010 CC bots
Post by lostcalpolydude on Jul 30^th, 2010, 4:17pm

on 07/30/10 at 11:55:28, Tuks wrote:

i liked the fact that the completion of the bot ladder was an accomplishment

It still is for the people that join the community (it was for me when I completed it a while ago, even getting to the point where I could handle fast and blitz games without panicking felt like an accomplishment). It's not like people just joining the community are going to be better in the same way that bots and humans that have been around for a while are improving.

Maybe there could be a second bot ladder if needed, but at that point it starts to seem more complicated.

Title: Re: ratings for 2010 CC bots
Post by Fritzlein on Jul 30^th, 2010, 9:40pm

I don't think the bot ladder is invalidated if it doesn't have the strongest version of the latest bot. It still is a significant accomplishment and a major benchmark.

On the other hand, I think it makes sense to overhaul the ladder every few years. Why not make the best ladder with the available material? In particular, instead of having four versions of every bot (P1, P2, Fast, Blitz), let's have fewer versions of each bot and more different developers/years. It's good for newcomers to be exposed to as great a variety of styles as possible.

For example, instead of SharpP1, SharpP2, SharpFast, and SharpBlitz, we could have PragmatictheoryP1, SharpP2, PragmatictheroyFast, and SharpBlitz. In addition to a wide variety of bot styles, it makes sense to have a wide range of ratings, from ArimaaScoreP1 to Marwin2010Blitz.

I think the ladder is OK as is, but would be improved by an overhaul.

Title: Re: ratings for 2010 CC bots
Post by Eltripas on Jul 30^th, 2010, 9:55pm

on 07/30/10 at 11:55:28, Tuks wrote:

I agree with Tuks, I want some new bots on the ladder.

Title: Re: ratings for 2010 CC bots
Post by 99of9 on Jul 31^st, 2010, 6:05am

on 07/29/10 at 17:41:16, Fritzlein wrote:

I agree with limiting bots to one core, and with disabling bots that don't give you the option to limit to one core. We should ask 99of9 about this; maybe there is a non-obvious flag in GnoBot 2010.

Omar emailed me, but sadly the answer is no. I recompile when I need to change the number of cores. You can tell I'm a non-professional can't you!

If it's important enough I can send omar an alternative binary compiled for one core. But Gnobot2010 is very very similar to Gnobot2009 anyway, so it might not be worth it.

Title: Re: ratings for 2010 CC bots
Post by Nombril on Aug 21^st, 2010, 3:09pm

I noticed that Marwin2010 Blitz can't be started? Unless I don't understand how to use the "Join game against..." command?

Title: Re: ratings for 2010 CC bots
Post by omar on Aug 21^st, 2010, 5:34pm

This bot can only be started and then the player joins the game it creates. It can't be asked to join a game a player creates since the time control could be set to something not considered Blitz.

I might have been testing something and forgot to reset the bot to start only.

Title: Re: ratings for 2010 CC bots
Post by Dolus on Nov 16^th, 2011, 5:38pm

on 07/29/10 at 00:00:33, omar wrote:

*bumping an old topic*

Is this information (CC, P1, etc.) something we can add to the bots ladder and/or advanced ladder pages? I've always wondered what the different extensions meant, and started gathering some of it based on game time options I've gathered or by perusing the forums, but it would be much easier if this was up front when picking which bot to play against.

P1 and P2 are probably the biggest things that need to be explained, but if an explanation for all of them could be listed, that would be great and very helpful.