Arimaa Forum - Print Page


    
      
        Arimaa Forum
        (http://arimaa.com/arimaa/forum/cgi/YaBB.cgi)
      

        Arimaa >> General Discussion >> Automatic bot games decreasing rating accuracy?
        
(Message started by: Fritzlein on May 7^th, 2006, 10:40am)

Title: Automatic bot games decreasing rating accuracy?
Post by Fritzlein on May 7^th, 2006, 10:40am

When the server first started having bots automatically play each other, I thought it was a great idea to increase the accuracy of the bot ratings. At the time, the low bots on the scale were mostly over-rated and the top bots were mostly under-rated. As predicted, the bots ratings have spread out quite a bit since then.

Unfortunately, there are several notable bugs in the bots:

1. All versions of Bomb seem to have a repetition bug which Clueless triggers. Bomb won't undo the opponent's move, but it will repeat position, and Clueless' shuffling style often earns it a win by repetition.

2. Clueless2006 thinks the game is over and plays randomly when the opponent has no rabbits, so super-weak bot opponents can win on material after losing all their rabbits in the opening.

3. Gnobot has a timeout bug around 250 moves that gets triggered against another defensive bot, in particular against Clueless.

Bots taking advantage of each others bugs makes the ratings more skewed, not more accurate.

Moreover, a lot of what has been pushing up the ratings of the top bots is not bots playing among themselves, but a newcomer getting stuck on a particular rung of the bot ladder. For example scienceman has 1 win and 25 losses against Clueless2005P1 and also 1 win and 32 losses against Bomb2005P1. That's not the only example of people shedding many points to a bot they can't beat. There are also some examples of harvesting points against a low bot that the player can beat: This is different from before when only top players pumped their ratings against bots, so no bot under 1750 was worth the effort of bashing. Bashing the low bots helps keep the rating scale spread out, whereas bashing the high bots kept it compressed.

Before there was the bot ladder, there were many bots that seldom got played by anyone, so inaccurate ratings took forever to self-correct. Nowadays the bot ladder is ensuring that none of the standard bots languish in obscurity. If one player messes up the rating of some bot, it won't be long before another person joins, works through the bot ladder, and provides some correction.

Given that the bot ladder is ensuring a greater variety of human opponents for each bot AND given that bot vs. bot games seem to trigger bugs which make ratings less accurate, I vote that the automatic bot vs. bot games be abolished.

Title: Re: Automatic bot games decreasing rating accuracy
Post by Arimanator on May 7^th, 2006, 12:54pm

What do you mean by automatic? Do these games happen at random?

Title: Re: Automatic bot games decreasing rating accuracy
Post by 99of9 on May 7^th, 2006, 8:46pm

Omar has a script that starts Bot v Bot games whenever the server is not very busy with bots.

Title: Re: Automatic bot games decreasing rating accuracy
Post by frostlad on May 7^th, 2006, 11:37pm

I think that the idea of bot v bot games and what it should accomplish is superb. I also think that on the whole the bot v bot games manages to keep the bots rated about correctly.

Yes there are bugs in some bots vs certain other bots. I think though that the games against the other bots serve to smooth out any irregularities from bugs.

Also, I think the most important thing to think about is not what the idea of the bot v bot games and the bot ladder do right now but if they continue and if the bugs of the bots are fixed by their trainers in next year's set of bots then these problems won't be there ( hopefully ) and then do we want bot v bot games?
I think that if without the bugs that the bot v bot games are good, that we should continue them anyway. I think the other games should serve to even it out enough, and hopefully next year's set of bots can solve some of those bugs.

Bot v Bot games also are interesting for me to see how the bots play against each other.

Title: Re: Automatic bot games decreasing rating accuracy
Post by 99of9 on May 8^th, 2006, 12:48am

I will fix Gnobby's 250 move bug before 2007 if someone reminds me in Nov/Dec. That will mean the Clueless games will be decided on score :o.

Title: Re: Automatic bot games decreasing rating accuracy
Post by Arimanator on May 8^th, 2006, 2:26am

on 05/07/06 at 20:46:04, 99of9 wrote:

Omar has a script that starts Bot v Bot games whenever the server is not very busy with bots.

I had my game crash from system overload several times now and I wonder if this isn't the cause of it. It would certainly be a bad thing if these random games should deter potential players.

Title: Re: Automatic bot games decreasing rating accuracy
Post by frostlad on May 8^th, 2006, 11:31am

Omar's script is set to start bot v bot only in the case that there is at most 1 game currently running on the server. So, the only way that the bot v bot games can contribute to system overload is if one of the games starts and then a bunch of people log on and start a game.
I honestly don't know how much this contributes to the server overload, but I know that Omar tried to account for that.