Arimaa Forum - Print Page


    
      
        Arimaa Forum
        (http://arimaa.com/arimaa/forum/cgi/YaBB.cgi)
      

        Arimaa >> General Discussion >> Rating lists based on year periods
        
(Message started by: sg4032 on Nov 2^nd, 2009, 12:52pm)

Title: Rating lists based on year periods
Post by sg4032 on Nov 2^nd, 2009, 12:52pm

Hi, I'm new to this site and I have only read the rules of Arimaa by now :) but I got interested in the rating management of the Arimaa community too.

I'm actually often spending time with computer chess, so that's where the idea comes from. As all the Arimaa game data is comfortably downloadable I decided to put all games divided by years into a rating calculation script. It's based on programs which are used for chess tournaments, mainly for chess engines (computers).

That's what I got. I'll only show excerpts from the first part.

The conditions were the following (can be changed):
- no Guest games were included
- only players with 30 or more games are shown
- at least 10 of the 30 games against different (unique) players
- average rating of all players (including the hidden players) is 1000

This one is of the year 2005 with 68 players overall in the list.
first 30:

http://img249.imageshack.us/img249/8848/2005b.png

score is (points divided by games)
av_opp is the average opponent rating
diffpl. is the unique opponent number

This is one list based on the games in the period 2008-11-01 to 2009-10-31 (last 12 months), 234 players overall in the list.
first 30:

http://img28.imageshack.us/img28/7089/200811200910.png

Maybe you can gain some ideas of that for your rating system. Such a list may be updated monthly and it represents the performance of the players based on the previous 12 months.

Edit:
Same as above (for the last 12 months) but no games with bots considered (it's the whole list):

http://img10.imageshack.us/img10/1020/200811200910nobots.png

Some info on win expectation depending on rating difference:

50 more points	57,1%
100 more points	64,0%
200 more points	76,0%
300 more points	84,9%
400 more points	90,9%
500 more points	94,7%
600 and more points	100%

Title: Re: Rating lists based on year periods
Post by Ice on Nov 2^nd, 2009, 5:23pm

Great analysis.

Does that mean that over the last 12 months, only 12 people have either played at least 30 HvH games and more than 10 unique players? That's rather interesting, and admittedly a little disappointing. ( I realize that I didn't make that list either. Hopefully that'll be different if we do this again 12 months from now )

Title: Re: Rating lists based on year periods
Post by sg4032 on Nov 3^rd, 2009, 3:55am

on 11/02/09 at 17:23:50, Ice wrote:

Does that mean that over the last 12 months, only 12 people have either played at least 30 HvH games and more than 10 unique players?

Exactly.

on 11/02/09 at 17:23:50, Ice wrote:

That's rather interesting, and admittedly a little disappointing. ( I realize that I didn't make that list either. Hopefully that'll be different if we do this again 12 months from now )

The list can be updated monthly. So it may already change after a month if you manage to play often enough in November to meet the conditions. :)

Title: Re: Rating lists based on year periods
Post by Fritzlein on Nov 3^rd, 2009, 6:02am

Thanks for posting a new rating list, sg4032. It is a project several of us have tried our hand at, and it is interesting to see how the different methodologies stack up against each other.

If I'm not mistaken, your ratings are calculated by taking winning percentage and average opponent ratings. Linearly averaging ratings can result in some weird rating behavior, because the expected wins curve is non-linear.

For example, suppose that I have two wins and two losses against players rated 2000. Naturally a 50% win rate against average opponents of 2000 gives me a rating of 2000 too. Then suppose I play a 1000-rated opponent and crush him. After that I have a 60% win rate against an average opposition of 1800, for a rating of 1870. Therefore, by beating another player I have lost 130 rating points!

The problem occurs in reverse as well. Someone with two wins and two losses against 1000-rated opponents can gain 130 rating points by losing to a 2000-rated opponent.

There is something fishy when winning a game can hurt your rating and losing a game can help your rating. It is an issue whenever people have an incentive to avoid certain opponents, and only seek out the opponents that they know will help boost their rating. Just something to consider...

Title: Re: Rating lists based on year periods
Post by sg4032 on Nov 3^rd, 2009, 7:33am

Hello Fritzlein,

you're right about that issue. I made my own thoughts about that and I think I've got a solution.

First, I think practically that problem doesn't affect all the ratings so much. At least the mid-ranked players will have a similar probablity to have played a much stronger as a much weaker opponent, so the fake performance ratings balance each other out. For the strongest and weakest players that's a problem indeed.

Now I thought of the following procedure. I'll let the script calculate the rating numbers, then look for each player, which are his opponents with the rating numbers out of the +-600-points-range (600 points + or - is what one gets for a 1 or 0 score) and then just delete the games out of the input games list. Then it'll just start over again. If necessesary, it can happen several times. At the end I'll have those games sorted out and only games between similar-strengh-opponents will stay. This would lead to less games but actually exactly those games are unnecessary for the rating evaluation.

Fortunately it's my own script so I can modify and improve it the way I want :). I'll try it out. Will take some time to implement...

Title: Re: Rating lists based on year periods
Post by Tuks on Nov 3^rd, 2009, 10:58am

d**n , i didnt realize i played so many humans

Title: Re: Rating lists based on year periods
Post by Fritzlein on Nov 3^rd, 2009, 11:35am

Yeah, Tuks, it looks like you played more games against humans than anyone else in the last twelve months, and more distinct humans than anyone except p4n1q.

Title: Re: Rating lists based on year periods
Post by sg4032 on Nov 5^th, 2009, 4:07pm

Hi again, I've got some news. :)

So the problems of the script approach were:

1) The Elo(Score) function is non-linear so the performance calculated by score reached against an "average opponent" is inaccurate (performance against 'mean value of opponent ratings' isn't equal to mean value of 'performances against opponents').

2) Also when considering opponent ratings out of the 600-points-range, the better player is disadvantaged even if he wins, the worse player profits from the game even when losing.

Now the solutions:

1)The easiest way to solve the first problem is taking a linear function.
There's one interesting article I found which deals with disadvantages of the Elo formula. The author finds out that linear prediction is even more accurate.
http://www.chessbase.com/newsdetail.asp?newsid=562
http://www.chessbase.com/images2/2002/sonas03.gif

At least I think it's not worse.

2) I described how I want to solve this one in my previous posting. It works. Some games get deleted but they're of no value anyway.

Here are the results:
I changed the conditions to less games required (15 games, 5 different opponents).

First the rating list based on the linear ("Sonas" as the author's name) formula.
Latest 12 months, no bots games.

http://img200.imageshack.us/img200/9086/200811200910nobotssonas.png

The good point of the linear formula is the simplicity when reading the ratings.
To find out the chances of winning: difference/10+50%.
E.g. winning chance of player2 against player1 is 112/10+50%=61,2%.
~~1000 points difference mean a 100% win percentage.~~ Sorry, 500 points difference mean a 100% win percentage.

Now the same list with dropped games between players with higher rating difference than 500.

http://img200.imageshack.us/img200/7677/sonas200811200910nobots.png

One can see the profits for the highest ranked players (the rating number and the average_opp. rating number). Some games had to be deleted but now the list is more accurate.

Summing up:

the rating list is based on games in a certain period (e.g. a year)
the list can be updated monthly (or even daily if necessary)
estimation of winning chance easy to calculate
the rating numbers are calculated based on the average rating number and the average score
the games between players, where one has a higher win percentage than 100% (lower than 0%) are deleted and the list is recalculated again (in several iterations, if necessary)
the average rating number of all players remains constant (e.g. 1000 like above)

Title: Re: Rating lists based on year periods
Post by Fritzlein on Nov 5^th, 2009, 5:31pm

The most notable difference between your ratings and woh's Whole History Ratings is that aaaa is much higher in your standings. I looked at his game record to try to discover the reason. My best guess is that you are including unrated games in your calculations. Aaaa has two victories over me this year from games that were 0/0/0/0/0 unrated analysis games. If my guess is correct, I'm curious to see what your ratings look like when they are based on rated HvH games only.

Title: Re: Rating lists based on year periods
Post by sg4032 on Nov 6^th, 2009, 7:07am

Thanks, my fault. I wondered why there was an archive called "rated games"... Now I changed the parsing algorithm.

Last 12 months, no bot games:
http://img442.imageshack.us/img442/2249/screenshotk.png

Following games were removed:
aaaa - Fritzlein 2-1
Libor - kiwik 1-0
zuborm - celbos 1-0
Eltripas - porkesi 9-2

The reason why players have even more games removed is, how I found out, the last added algorithm with dropping games with players with too different playing strengh. In the case of Fritzlein the removed games led to a higher rating and more games against weak players were removed (11 players). So it makes sense.

Which database was used for woh's rating list you are referring to? Maybe I should just feed the script with exactly the same database for comparing.

Title: Re: Rating lists based on year periods
Post by woh on Nov 6^th, 2009, 8:26am

on 11/06/09 at 07:07:48, sg4032 wrote:

Which database was used for woh's rating list you are referring to? Maybe I should just feed the script with exactly the same database for comparing.

The allgames files from the Arimaa Games Archive are used for the WHR rankings. All rated HvH games are considered.

Title: Re: Rating lists based on year periods
Post by sg4032 on Nov 6^th, 2009, 10:31am

Ok, so this rating works similar to official chess' where after each game the players get new rankings immediately. Such systems of course work best with as much games as possible but it needs a start rating at the beginning which is equal for all players (so at the beginning the ratings are most incorrect but become more accurate with games number increasing).
For my script many games don't make sense because all games count equally and older games don't represent the current playing strengh accurate, so "last year games" as a database are a good compromise. The advantage is that it works more precise with few games and with new players. It would suit best for monthly leagues or tournaments without a certain schedule where players can play anyone they meet during the tournament duration (with conditions to have a certain amount of games and diff. players).

The whole rating list (same as above) with all players and no conditions to meet (klick on green download button):
http://www.file-upload.net/download-1994018/allgames200811-200910_nobots.xls.html

Edit: I realised that dropping games with player's rating difference > 499.9 excludes players with score 1 or 0. >500 is the right condition.
http://www.file-upload.net/download-1994121/allgames200811-200910_nobots2.xls.html