Author |
Topic: Rating lists based on year periods (Read 941 times) |
|
sg4032
Forum Junior Member
Gender: 
Posts: 6
|
 |
Rating lists based on year periods
« on: Nov 2nd, 2009, 12:52pm » |
 Quote  Modify
|
Hi, I'm new to this site and I have only read the rules of Arimaa by now  but I got interested in the rating management of the Arimaa community too.
I'm actually often spending time with computer chess, so that's where the idea comes from. As all the Arimaa game data is comfortably downloadable I decided to put all games divided by years into a rating calculation script. It's based on programs which are used for chess tournaments, mainly for chess engines (computers).
That's what I got. I'll only show excerpts from the first part.
The conditions were the following (can be changed):
- no Guest games were included
- only players with 30 or more games are shown
- at least 10 of the 30 games against different (unique) players
- average rating of all players (including the hidden players) is 1000
This one is of the year 2005 with 68 players overall in the list.
first 30:
score is (points divided by games)
av_opp is the average opponent rating
diffpl. is the unique opponent number
This is one list based on the games in the period 2008-11-01 to 2009-10-31 (last 12 months), 234 players overall in the list.
first 30:
Maybe you can gain some ideas of that for your rating system. Such a list may be updated monthly and it represents the performance of the players based on the previous 12 months.
Edit:
Same as above (for the last 12 months) but no games with bots considered (it's the whole list):
Some info on win expectation depending on rating difference:
| 50 more points | 57,1% | | 100 more points | 64,0% | | 200 more points | 76,0% | | 300 more points | 84,9% | | 400 more points | 90,9% | | 500 more points | 94,7% | | 600 and more points | 100% |
|
| « Last Edit: Nov 3rd, 2009, 8:27am by sg4032 » |
 IP Logged |
|
|
|
Ice
Forum Full Member
Arimaa player #3201
Gender:
Posts: 25
|
|
Re: Rating lists based on year periods
« Reply #1 on: Nov 2nd, 2009, 5:23pm » |
Quote Modify
|
Great analysis.
Does that mean that over the last 12 months, only 12 people have either played at least 30 HvH games and more than 10 unique players? That's rather interesting, and admittedly a little disappointing. ( I realize that I didn't make that list either. Hopefully that'll be different if we do this again 12 months from now )
|
|
IP Logged |
|
|
|
sg4032
Forum Junior Member
Gender:
Posts: 6
|
|
Re: Rating lists based on year periods
« Reply #2 on: Nov 3rd, 2009, 3:55am » |
Quote Modify
|
on Nov 2nd, 2009, 5:23pm, Ice wrote:| Does that mean that over the last 12 months, only 12 people have either played at least 30 HvH games and more than 10 unique players? |
|
Exactly.
on Nov 2nd, 2009, 5:23pm, Ice wrote:| That's rather interesting, and admittedly a little disappointing. ( I realize that I didn't make that list either. Hopefully that'll be different if we do this again 12 months from now ) |
|
The list can be updated monthly. So it may already change after a month if you manage to play often enough in November to meet the conditions.
|
|
IP Logged |
|
|
|
Fritzlein
Forum Guru
Arimaa player #706
Gender:
Posts: 5928
|
|
Re: Rating lists based on year periods
« Reply #3 on: Nov 3rd, 2009, 6:02am » |
Quote Modify
|
Thanks for posting a new rating list, sg4032. It is a project several of us have tried our hand at, and it is interesting to see how the different methodologies stack up against each other.
If I'm not mistaken, your ratings are calculated by taking winning percentage and average opponent ratings. Linearly averaging ratings can result in some weird rating behavior, because the expected wins curve is non-linear.
For example, suppose that I have two wins and two losses against players rated 2000. Naturally a 50% win rate against average opponents of 2000 gives me a rating of 2000 too. Then suppose I play a 1000-rated opponent and crush him. After that I have a 60% win rate against an average opposition of 1800, for a rating of 1870. Therefore, by beating another player I have lost 130 rating points!
The problem occurs in reverse as well. Someone with two wins and two losses against 1000-rated opponents can gain 130 rating points by losing to a 2000-rated opponent.
There is something fishy when winning a game can hurt your rating and losing a game can help your rating. It is an issue whenever people have an incentive to avoid certain opponents, and only seek out the opponents that they know will help boost their rating. Just something to consider...
|
|
IP Logged |
|
|
|
sg4032
Forum Junior Member
Gender:
Posts: 6
|
|
Re: Rating lists based on year periods
« Reply #4 on: Nov 3rd, 2009, 7:33am » |
Quote Modify
|
Hello Fritzlein,
you're right about that issue. I made my own thoughts about that and I think I've got a solution.
First, I think practically that problem doesn't affect all the ratings so much. At least the mid-ranked players will have a similar probablity to have played a much stronger as a much weaker opponent, so the fake performance ratings balance each other out. For the strongest and weakest players that's a problem indeed.
Now I thought of the following procedure. I'll let the script calculate the rating numbers, then look for each player, which are his opponents with the rating numbers out of the +-600-points-range (600 points + or - is what one gets for a 1 or 0 score) and then just delete the games out of the input games list. Then it'll just start over again. If necessesary, it can happen several times. At the end I'll have those games sorted out and only games between similar-strengh-opponents will stay. This would lead to less games but actually exactly those games are unnecessary for the rating evaluation.
Fortunately it's my own script so I can modify and improve it the way I want . I'll try it out. Will take some time to implement...
|
|
IP Logged |
|
|
|
Tuks
Forum Guru
Arimaa player #2626
Gender:
Posts: 203
|
|
Re: Rating lists based on year periods
« Reply #5 on: Nov 3rd, 2009, 10:58am » |
Quote Modify
|
d**n
, i didnt realize i played so many humans
|
|
IP Logged |
|
|
|
Fritzlein
Forum Guru
Arimaa player #706
Gender:
Posts: 5928
|
|
Re: Rating lists based on year periods
« Reply #6 on: Nov 3rd, 2009, 11:35am » |
Quote Modify
|
Yeah, Tuks, it looks like you played more games against humans than anyone else in the last twelve months, and more distinct humans than anyone except p4n1q.
|
|
IP Logged |
|
|
|
sg4032
Forum Junior Member
Gender:
Posts: 6
|
|
Re: Rating lists based on year periods
« Reply #7 on: Nov 5th, 2009, 4:07pm » |
Quote Modify
|
Hi again, I've got some news.
So the problems of the script approach were:
1) The Elo(Score) function is non-linear so the performance calculated by score reached against an "average opponent" is inaccurate (performance against 'mean value of opponent ratings' isn't equal to mean value of 'performances against opponents').
2) Also when considering opponent ratings out of the 600-points-range, the better player is disadvantaged even if he wins, the worse player profits from the game even when losing.
Now the solutions:
1)The easiest way to solve the first problem is taking a linear function.
There's one interesting article I found which deals with disadvantages of the Elo formula. The author finds out that linear prediction is even more accurate.
http://www.chessbase.com/newsdetail.asp?newsid=562
At least I think it's not worse.
2) I described how I want to solve this one in my previous posting. It works. Some games get deleted but they're of no value anyway.
Here are the results:
I changed the conditions to less games required (15 games, 5 different opponents).
First the rating list based on the linear ("Sonas" as the author's name) formula.
Latest 12 months, no bots games.
The good point of the linear formula is the simplicity when reading the ratings.
To find out the chances of winning: difference/10+50%.
E.g. winning chance of player2 against player1 is 112/10+50%=61,2%.
1000 points difference mean a 100% win percentage. Sorry, 500 points difference mean a 100% win percentage.
Now the same list with dropped games between players with higher rating difference than 500.
One can see the profits for the highest ranked players (the rating number and the average_opp. rating number). Some games had to be deleted but now the list is more accurate.
Summing up:
- the rating list is based on games in a certain period (e.g. a year)
- the list can be updated monthly (or even daily if necessary)
- estimation of winning chance easy to calculate
- the rating numbers are calculated based on the average rating number and the average score
- the games between players, where one has a higher win percentage than 100% (lower than 0%) are deleted and the list is recalculated again (in several iterations, if necessary)
- the average rating number of all players remains constant (e.g. 1000 like above)
|
| « Last Edit: Nov 6th, 2009, 4:52am by sg4032 » |
IP Logged |
|
|
|
Fritzlein
Forum Guru
Arimaa player #706
Gender:
Posts: 5928
|
|
Re: Rating lists based on year periods
« Reply #8 on: Nov 5th, 2009, 5:31pm » |
Quote Modify
|
The most notable difference between your ratings and woh's Whole History Ratings is that aaaa is much higher in your standings. I looked at his game record to try to discover the reason. My best guess is that you are including unrated games in your calculations. Aaaa has two victories over me this year from games that were 0/0/0/0/0 unrated analysis games. If my guess is correct, I'm curious to see what your ratings look like when they are based on rated HvH games only.
|
|
IP Logged |
|
|
|
sg4032
Forum Junior Member
Gender:
Posts: 6
|
|
Re: Rating lists based on year periods
« Reply #9 on: Nov 6th, 2009, 7:07am » |
Quote Modify
|
Thanks, my fault. I wondered why there was an archive called "rated games"... Now I changed the parsing algorithm.
Last 12 months, no bot games:
Following games were removed:
aaaa - Fritzlein 2-1
Libor - kiwik 1-0
zuborm - celbos 1-0
Eltripas - porkesi 9-2
The reason why players have even more games removed is, how I found out, the last added algorithm with dropping games with players with too different playing strengh. In the case of Fritzlein the removed games led to a higher rating and more games against weak players were removed (11 players). So it makes sense.
Which database was used for woh's rating list you are referring to? Maybe I should just feed the script with exactly the same database for comparing.
|
| « Last Edit: Nov 6th, 2009, 7:08am by sg4032 » |
IP Logged |
|
|
|
woh
Forum Guru
Arimaa player #2128
Gender:
Posts: 254
|
|
Re: Rating lists based on year periods
« Reply #10 on: Nov 6th, 2009, 8:26am » |
Quote Modify
|
on Nov 6th, 2009, 7:07am, sg4032 wrote:| Which database was used for woh's rating list you are referring to? Maybe I should just feed the script with exactly the same database for comparing. |
|
The allgames files from the Arimaa Games Archive are used for the WHR rankings. All rated HvH games are considered.
|
|
IP Logged |
|
|
|
sg4032
Forum Junior Member
Gender:
Posts: 6
|
|
Re: Rating lists based on year periods
« Reply #11 on: Nov 6th, 2009, 10:31am » |
Quote Modify
|
Ok, so this rating works similar to official chess' where after each game the players get new rankings immediately. Such systems of course work best with as much games as possible but it needs a start rating at the beginning which is equal for all players (so at the beginning the ratings are most incorrect but become more accurate with games number increasing).
For my script many games don't make sense because all games count equally and older games don't represent the current playing strengh accurate, so "last year games" as a database are a good compromise. The advantage is that it works more precise with few games and with new players. It would suit best for monthly leagues or tournaments without a certain schedule where players can play anyone they meet during the tournament duration (with conditions to have a certain amount of games and diff. players).
The whole rating list (same as above) with all players and no conditions to meet (klick on green download button):
http://www.file-upload.net/download-1994018/allgames200811-200910_nobots .xls.html
Edit: I realised that dropping games with player's rating difference > 499.9 excludes players with score 1 or 0. >500 is the right condition.
http://www.file-upload.net/download-1994121/allgames200811-200910_nobots 2.xls.html
|
| « Last Edit: Nov 6th, 2009, 11:21am by sg4032 » |
IP Logged |
|
|
|
|
Home
Help
Search
Members
Login
Register
Reply
Notify of replies
Send Topic
Print