Arimaa Forum - Print Page


    
      
        Arimaa Forum
        (http://arimaa.com/arimaa/forum/cgi/YaBB.cgi)
      

        Arimaa >> General Discussion >> Does bots play unrated games differently ?
        
(Message started by: Tachyon on Sep 27^th, 2008, 12:40am)

Title: Does bots play unrated games differently ?
Post by Tachyon on Sep 27^th, 2008, 12:40am

I have noticed that bots that I have beaten in unrated games does not change position on the bot ladder ... to below me.

Is there a reason for this ??

Title: Re: Does bots play unrated games differently ?
Post by Fritzlein on Sep 27^th, 2008, 3:42am

Yes, to pass a bot on the bot ladder, you must beat it in a rated game.

There is a reason for distinguishing rated and unrated games. The games database is used for a variety of purposes, including analyzing the relative values of the pieces, building an opening book for computer programs, estimating whether Gold or Silver has an advantage, etc.

When doing any kind of serious analysis, people exclude unrated games, because unrated games are often inapplicable for a variety of reasons. For example, the game might have been a teaching game between friends, or it might have been played at a handicap, or it might have been played in a crazy style to try to break a bot-bashing record at the risk of losing.

The distinction between rated and unrated games is not always so precise. Some handicap and experimental games end up rated, while some hard-fought regular games are unrated. Nevertheless, the distinction is real. (Indeed your question proves there is a distinction in your mind; there is a reason you play some games rated and some unrated.) People do in general play more seriously when the games are rated, because they feel that they have something on the line. I think Omar wants the bot ladder games to be in the serious category, where you had something to lose and something to gain, rather than unrated games where it is more likely something wacky was going on.

Title: Re: Does bots play unrated games differently ?
Post by chessandgo on Sep 27^th, 2008, 7:58am

... and at least one version of GnoBot does indeed play differently in rated and unrated games.

Title: Re: Does bots play unrated games differently ?
Post by omar on Sep 27^th, 2008, 11:30am

Also I think Clueless setups differently for unrated games.

Yes, it is possible for a bot to play differently based on if the game is rated or not. Also the game state file that the bot gets as input provides pretty much all the information about the game that a human would have; including the name of their opponent. So it is even possible for a bot to play differently against different players; or play differently against humans than it does against bots.

Title: Re: Does bots play unrated games differently ?
Post by Tachyon on Sep 27^th, 2008, 1:15pm

Quote:

When doing any kind of serious analysis, people exclude unrated games, because unrated games are often inapplicable for a variety of reasons...

This assumes that rated games are played in a controlled environment and with the kind of oversight that ensures or at least promotes a reliable rating e.g. in official chess tournaments ( which is not the case on the arimaa platform. )

Quote:

... or it might have been played in a crazy style to try to break a bot-bashing record at the risk of losing ...

I did in fact play a number of such-bot bashing games once I got a little more proficient in the game with actually relatively little risk of losing due to the predictable play of bots. As a consequence my rating has rapidly increased to an unrealistic level ( from 1228 to 1707 ). In fact I am sure I can push my rating to over 2000 if I chose to do so ( There is however little point ... it merely demonstrate that the current implementation of the rating system is not truly reflective of a players strength. )

Quote:

People do in general play more seriously when the games are rated.

Only if they take the game seriously.
Most new players just play for fun and only a percentage of them go on to play the game with any significance to rating.

I was not even aware that I had the option of playing unrated games during my first 60+ games. There was a lot of those games that I played purely for the purpose of teaching myself the game. I did not particularly care whether I won or lost the game neither did I pay attention to the rating.

Title: Re: Does bots play unrated games differently ?
Post by Fritzlein on Sep 27^th, 2008, 3:09pm

on 09/27/08 at 13:15:56, Tachyon wrote:

No, it only assumes that rated games are more reliable than unrated games. I'm definitely not saying that there is a controlled environment, or that the ratings are accurate. I know you have played record-setting attempts rated. I'm well aware that the game room ratings are inaccurate due to bot bashing and other factors. But the point is that there is a difference between rated and unrated games.

on 09/27/08 at 13:15:56, Tachyon wrote:

Most new players just play for fun and only a percentage of them go on to play the game with any significance to rating

This is not at all accurate in my experience. Many new players are scared to play rated games, because they think then the game will "count" somehow. As you have noted, it isn't obvious that an option to play unrated games even exists, but just look how many newcomers go out of their way to find that option and use it.

Perhaps you personally have never cared about your rating, but other people do, for the most part, treat a game differently depending on whether or not it is rated.

Anyway, since you don't care about your rating, it is no problem for you to play up the bot ladder in rated mode, right?

Title: Re: Does bots play unrated games differently ?
Post by Arimabuff on Sep 27^th, 2008, 10:30pm

on 09/27/08 at 15:09:20, Fritzlein wrote:

...Anyway, since you don't care about your rating, it is no problem for you to play up the bot ladder in rated mode, right?

He evidently cares enough to start a thread about it. Tachyon it looks to me like you like to talk in circles.

Title: Re: Does bots play unrated games differently ?
Post by Tachyon on Sep 27^th, 2008, 10:35pm

Fritzlein .... No I do not mind playing the bot ladder in rated mode ...
I was just curious as to the reason why this is required.

I seem to have made the mistaken assumption that the board community is interested in having the players rating serve the purpose of reflecting player's relative strengths as accurately as possible.

Since this is not the case and the community is aware and happy with the existence of inaccurate ratings, my comments above are largely irrelevant.

Title: Re: Does bots play unrated games differently ?
Post by Tachyon on Sep 27^th, 2008, 10:47pm

Quote:

He evidently cares enough to start a thread about it. Tachyon it looks to me like you like to talk in circles.

Arimabuff ... if you bothered to read this thread properly then you will see I started this thread as a simple question about how rated vs unrated games affect the bot ladder, not about the value or relevance of my rating.

If you further read my text properly then you will notice that I refer to the first 60+ games that I played largely for the purpose of learning the game and that I nowhere stated that I attach no value to a meaningful rating.

Title: Re: Does bots play unrated games differently ?
Post by omar on Sep 27^th, 2008, 11:19pm

on 09/27/08 at 13:15:56, Tachyon wrote:

In fact I am sure I can push my rating to over 2000 if I chose to do so ( There is however little point ... it merely demonstrate that the current implementation of the rating system is not truly reflective of a players strength. )

Yes it is quite difficult to have a reliable rating system when you also allow the players to pick their opponents. The standard method of calculating ratings doesn't hold up very well under these conditions. We've had a lot of discussions in the past about rating systems and how to improve them. It is still a work in progress.

There is also a P8 rating system which seems to provide more reliable ratings when players are allowed to pick their opponents. That system currently says you are rated around 1450 instead of 1700.

http://arimaa.com/arimaa/rating/testRatings/top_p8.html

I think you would find it quite challenging to reach a 2000 p8 rating while keeping the magnitude of the +-RUs under 100. Not that you couldn't do it if you were persistent, but just that you could not easily fool the p8 system and get there quickly by doing a lot of bot bashing.

aaaa has also been working on trying to make a more reliable rating system; I wonder what it says for your rating.

Title: Re: Does bots play unrated games differently ?
Post by Tachyon on Sep 28^th, 2008, 12:15am

Thanks Omar

May I ask why it is so important to let players pick their own opponents for rated games ?

Why not let games where players pick their own oponents be unrated and only rate games that are played in some form of organized tournament style ... or designate the current rating as an "unofficial rating" and create a new "official rating" just for tournament style games ?"

( "Tournament style" meaning that games are played under some form of oversight designed to promote the reliability of ratings.)

I believe that one will have more success by adressing the core issue rather that trying to fix the data using some kind of complex system.

Title: Re: Does bots play unrated games differently ?
Post by Fritzlein on Sep 28^th, 2008, 6:47am

on 09/27/08 at 22:35:33, Tachyon wrote:

I seem to have made the mistaken assumption that the board community is interested in having the players rating serve the purpose of reflecting player's relative strengths as accurately as possible.

Since this is not the case and the community is aware and happy with the existence of inaccurate ratings, my comments above are largely irrelevant.

I would say your first assumption was correct. The Arimaa community would like ratings to reflect the relative strengths of the players as accurately as possible.

I am baffled about the conclusion that because I (we?) want bot ladder games to be rated, we are happy with inaccurate ratings. If instead we had both rated and unrated games counted on the bot ladder, would that somehow show our commitment to accurate ratings? I fail to see how you make this inference. It seems to me you asked a question about one thing (bot ladder games being rated), and then started talking about something else (ratings being accurate), and pinned an attitude on me (that I don't care about inaccurate ratings) that doesn't follow from anything I have said, and that isn't true.

If you would read back in the forum, you would find that I have posted at great length about improving the accuracy of our rating system. I have probably devoted more time to the topic than anybody else here, so your comments sting a bit.

At first I was a big proponent of changing the rating formula in complicated ways to compensate for the fact that players can select their opponents. The general idea is to not count all games equally, i.e. to somehow make repeated plays against the same opponent count for less. Omar has implemented this general idea in the p8 ratings, which are more accurate than the game room ratings, but still not very accurate.

Another "complexifying" proposal of mine was to have bot games have a steeper expectation curve. Instead of the formula being E = 1/(1+((10^(R_a - R_b)/400))), for games involving a bot it could be E = 1/(1+((10^(R_a - R_b)/200))). This would limit the ill-gotten gains of bot-bashing to half of what is possible now, but I recognize it is only a band-aid.

Over time, I have come to believe that the only solution is something akin to what you suggest, i.e. only counting games where the players are not allowed to pick their opponents, and potentially also only games between humans. It's hard for mathematics to provide a fix when the underlying reality is what is broken. I expect that if we discussed what makes a rating system accurate you would find that we agree more than we disagree.

Title: Re: Does bots play unrated games differently ?
Post by Tachyon on Sep 28^th, 2008, 7:16am

Fritzlein ....

First of all my apologies if any of my postings offended you or anyone else for that matter. That was certainly not the intention.

Secondly ... please bear in mind that discussions on a forum tend to happen in "slow motion". In a real life situation when someone make an assertion ... it is easy for the other party to interrupt and say ... "No, that is not what I meant and correct the assumption made". This allows the conversation to move on with an immediate clear understanding.

It unfortunately does not quite work that way in forums like this. e.g. You inferrred from my postings "since you don't care about your rating" which is incorrect since I never said that. I have tried to correct that impression in my response to Arimabuff.

Lastly ... I expect you are correct in that we will agree more than disagree if we were able to discuss this topic face to face.

Back to the thread Topic:

Given that my initial assumption is correct it seems that there is some motivation for the Arimaa community to pull together and resolve the rating issue. It would be great to have ratings that one can have some measure of confidence in.

It has also crossed my mind that as Arimaa becomes more mainstream the rating system will eventually come under scrutiny from others outside of the immediate Arimaa community and it would be preferable to ensure the system can stand up to such scrutiny before that time comes.

Title: Re: Does bots play unrated games differently ?
Post by Fritzlein on Sep 28^th, 2008, 8:22am

Likewise, I am sorry if I offended you, and that I jumped to an incorrect conclusion that you attach no importance to your rating. You are right that Internet discussions are trickier than face-to-face discussions.

There have been many threads about improving the rating system in the past, but I think there is plenty of room for more. Why don't you lay out your proposals in a new thread with a new title, and we'll go from there?

Title: Re: Does bots play unrated games differently ?
Post by omar on Sep 28^th, 2008, 11:27pm

on 09/28/08 at 00:15:46, Tachyon wrote:

May I ask why it is so important to let players pick their own opponents for rated games ?

Why not let games where players pick their own oponents be unrated and only rate games that are played in some form of organized tournament style ... or designate the current rating as an "unofficial rating" and create a new "official rating" just for tournament style games ?"

( "Tournament style" meaning that games are played under some form of oversight designed to promote the reliability of ratings.)

I believe that one will have more success by adressing the core issue rather that trying to fix the data using some kind of complex system.

Good question. Freedom of choice I guess and also to allow immeadiate gratification :-) We also don't have many "tournament style" games going on and a majority of the players don't have time to participate in such events due to scheduling constraints. Thus we would not accumulate many rated games. So perhaps our ratings would still not be very accurate but due to a different reason of not having enough rated games.

What we are really trying to do here is sheild our rating system from getting bad games as input. Our rating formulas are pretty good and given a good set of game they will produce good ratings. I spent a lot of time once experimenting with rating formulas. See the README and results file here:
http://arimaa.com/arimaa/rating/030423/

The basic Elo formula is a good model and there is not much improvement to be gained by trying to change it. Rather better rating accuracy can be gained by making sure good games are used to determine the ratings and not letting the rating system be exposed to abuse like pumping and sandbagging.

Using only games from events where the players were paired and did not have any choice in selecting their opponent is of course the best way to shield the rating system from getting bad games. But I think it may be a bit over-restrictive and in a fledgling game like Arimaa there will not be enough such games to make the ratings accurate. It takes about 95 games for a players rating to reach an accuracy of +-30 points. Even Fritzlein who has played over 500 rated HH games has only played 70 such event games. I have only 64 such games.

So to maximally use the games that we've accumulated requires having a complex layer that can intelligently decide which games are good to use and which aren't. This kind of starts to become a bit of an AI problem. But I think it is doable. Essentially it boils down to making use of all the observable data we have about a game (including game histories of the two players) to build filtering rules to shield the rating system from bad games. Or a more general way to do it is to weigh how much each game will count.

The resulting systems show be compared against each other not just on properties such as rating drift and convergence (see the README file from the link above), but also on how well they handle abusive behavior such as pumping and sandbagging.

So ideally that's the kind of rating system that I am shooting for in the Arimaa gameroom. But as usual never get enough time to work on it. If anyone else has an interest in this I would be happy to share whatever I have.

If a particular event requires specialized ratings such as using only HH games or giving more weight to postal or slow games we can always run that offline using the games database.

Title: Re: Does bots play unrated games differently ?
Post by Tachyon on Oct 1^st, 2008, 11:36pm

Thanks for the lengthy reply Omar.

Your point regarding the number of games it takes to build an accurate rating is something I haven't considered. It seems to me then that one is left with the following options:

1) Give the community the ability to play rated games and try to obtain reasonable ratings by means of

a) Identifying and Managing those factors that contribute to a skew in the ratings and
b) Using the rating calculation method to adjust ratings where possible.

2) Create more opportunities where games can be played in an organized tournament environment for new and experienced players alike to fast track the establishment of accurate ratings.

I also think it is important that the playing community is aware of the fact the ratings is a work in progress and that there is a longer term plan in place to stabilize the ratings. Maybe create a page that outline the rating system used and the issues being dealt with.

One point I would like to make is that it is quite difficult for new players to find enough games to play against other new players, since this relies heavily on enough players being in the gameroom at the same time. I have been in the game room quite a lot over the last month and not very often do I see players with a rating of sub 1600 that is asking for an invite.

I have also found that responding to an invite is often met with no response. I assume the player has issued an invite, got tired of waiting and left.

I am quite sure that if you would look at the stats you would find that the majority of players play far more HvB games that HvH games, partly for this reason. The only HvH games I have played was with far more experienced players that myself.

Title: Re: Does bots play unrated games differently ?
Post by Tachyon on Oct 3^rd, 2008, 11:14am

By the way , I am convinced bot_clueless2007P1 play unrated games differently ... It certainly changes creates different opening setups for rated games and seems to play more agressively.