Author |
Topic: Arimaa rating deflation (Read 30017 times) |
|
Fritzlein
Forum Guru
Arimaa player #706
Gender: 
Posts: 5928
|
 |
Re: Arimaa rating deflation
« Reply #105 on: Apr 3rd, 2006, 3:40pm » |
 Quote  Modify
|
We all know how lots of noobs join the server, lose a few games to low bots, and never come back. They tend to donate 30 to 100 rating points each. Those points stay among permanent players and contribute to rating inflation.
However, I wanted to study the effect of someone like Swynndla, who joins the server and loses a ton of points, but later wins them back with interest. Between server games 25572 and 26260, Swynndla reduced his RU from 120 to 30 by playing 71 rated games. He started at a rating of 1500 and ended at 1565, but that doesn't mean he robbed a net of 65 points from his opponents, because a variable RU means his rating was changing at a different speed than the ratings of his opopnents. In particular, on his way down to a rating of 1024, his rating was moving faster than when he clawed his way back up.
I posted ealier my conviction that folks like Swynndla have a deflationary effect beyond the traditional one of leaving at a higher rating than they join. (Heaven forbid Swynndla should leave soon, but when he does, he'll have a higher rating than 1500 where he started, so net points will leave the system with him.) When I actually ran the numbers on these 71 games, it showed that my theory doens't work. In fact, Swynndla actually only robbed 63 points from his opponents, i.e. less than the 65 he gained!
Where did my theory go wrong? Hmmm...
Actually, it partially worked. After 31 games, the point at which Swynndla first resurfaced above a 1500 rating, he had robbed his opponents of 66 points while only going up to 1512. That's a net deflation effect of 54 points. However, at that point his RU was still high at 70, and in subesquent games Swynndla peaked at rating of 1669 before falling back to 1565. He reversed the effect I predicted by gaining points at higher RU and losing back points at lower RU for a net inflation.
Furthermore, anyone who gains rating points above 1500 while they have a high RU, even if they only hold steady thereafter until their RU reaches 30, will gain more points in their own rating than they robbed from others to get there.
I guess I need to revise my deflation theory, and say instead that noobs almost universally contribute to inflation. Either they quickly cash out at a lower rating than 1500, which donates points to the system, or they stick around and start rising above 1500 before their RU bottoms out at 30, which means their own rating represents extra points in the system. The (much rarer) deflationary cases are noobs who make it down to RU 30 before pumping their rating back over 1500. Also there is traditional deflation from strong, established players leaving and taking their high rating along, but that doesn't happen much, because Arimaa is just too addictive. I note that our two highest-rated depatures, mouse and Arimanator, have each recently come back.  That leaves only OmarFast and bleitner as high-rated inactive players.
To add even more fuel to the inflationary fire, the continual bot vs. bot games are taking inflationary points away from the bottom few bots and spreading them much more quickly across the entire system. This drives those bottom bots back down in rating, so that they can rob noobs of points all the more quickly. Throw in the fact that we had a flood of noobs in February and early March, and I'd say we're probably presently experiencing rating inflation like never before.
|
|
 IP Logged |
|
|
|
frostlad
Forum Senior Member
Arimaa player #1704
Gender:
Posts: 46
|
|
Re: Arimaa rating deflation
« Reply #106 on: Apr 3rd, 2006, 11:21pm » |
Quote Modify
|
What does a player like filerank do to the point pool? just curious.
|
|
IP Logged |
|
|
|
Swynndla
Forum Guru
Arimaa player #1821
Posts: 235
|
|
Re: Arimaa rating deflation
« Reply #107 on: Apr 4th, 2006, 6:22am » |
Quote Modify
|
on Apr 3rd, 2006, 3:40pm, Fritzlein wrote:Heaven forbid Swynndla should leave soon
...
...
but that doesn't happen much, because Arimaa is just too addictive |
|
Indeed, I'm addicted and therefore I won't be leaving 
|
|
IP Logged |
|
|
|
Fritzlein
Forum Guru
Arimaa player #706
Gender:
Posts: 5928
|
|
Re: Arimaa rating deflation
« Reply #108 on: Apr 4th, 2006, 1:33pm » |
Quote Modify
|
on Apr 3rd, 2006, 11:21pm, frostlad wrote:| What does a player like filerank do to the point pool? just curious. |
|
While filerank was winning his first 42 rated games in a row, he boosted his rating 424 points to 1924 while robbing 207 points from his opponents, for a net inflation effect of 217 points. His RU was still 60 at that point. Of course, if he had quit and gone home at that point it would have been a 207 point contributor to deflation, but he didn't quit.
Over filerank's first 166 rated games total, he gained 276 net rating points to land at 1776. His opponents in that span gained a net 3 rating points. Therefore, even if filerank now takes his high rating and goes home, he has still contributed three points to rating inflation! If he stays around, however, he has contributed 279 points, mostly within his own rating.
As long as I'm at it, Frostlad, I might as well do your record too. You are a bit different from either Swynndla or filerank, because you had neither a huge drop nor a huge rise at the beginning. Instead you've sort of worked your way up slowly and steadily. In your first 121 rated games you gained 123 rating points while robbing 162 from your opponents. So unlike the other two, you've actually contributed to deflation to the tune of 39 points. Strange, eh?
I note that you have a better rate of gain (one point per game) than I do. I've played 798 games and only gained 778 rating points. 
|
| « Last Edit: Apr 4th, 2006, 1:57pm by Fritzlein » |
IP Logged |
|
|
|
Fritzlein
Forum Guru
Arimaa player #706
Gender:
Posts: 5928
|
|
Re: Arimaa rating deflation
« Reply #109 on: Apr 4th, 2006, 2:15pm » |
Quote Modify
|
Still in the same vein, I have 2997 rated HvB games in my database for 2006. (I am suspicious of the "rated" flag's accuracy, as I posted in another thread, but for now pretend it is correct.) In those 2997 games, the bots collectively gained 1448 points while the humans collectively lost 9687 points. It fits with the bigger picture.
[EDIT] I withdraw my innuendos against the rated flag. It was an error in one of my queries (not the one in this post) that made me unjustly suspicious.
|
| « Last Edit: Apr 6th, 2006, 12:43am by Fritzlein » |
IP Logged |
|
|
|
frostlad
Forum Senior Member
Arimaa player #1704
Gender:
Posts: 46
|
|
Re: Arimaa rating deflation
« Reply #110 on: Apr 5th, 2006, 10:30am » |
Quote Modify
|
Do you have a program or what is the formula that you use to calculate the net gained points vs how many I took fritz?
|
|
IP Logged |
|
|
|
Fritzlein
Forum Guru
Arimaa player #706
Gender:
Posts: 5928
|
|
Re: Arimaa rating deflation
« Reply #111 on: Apr 5th, 2006, 1:04pm » |
Quote Modify
|
(1) I download the zipped game files from at text from http://arimaa.com/arimaa/download/gameData/
(2) I import the data into Microsoft Access
(3) I query all rated games of one player, to output columns PlayerRating, OppRating, PlayerRU, OppRU, PlayerScore.
(4) I cut and paste the query results into an Microsoft Excel spreadsheet with formulas to calculate, for each game, how many rating points Player and Opp gained and lost. I also have columns for cumulative loss/gain.
If I were a programmer, I would automate this process somehow, but given my limited abilities I just kludge along the best I can.
|
|
IP Logged |
|
|
|
omar
Forum Guru
Arimaa player #2
Gender:
Posts: 1003
|
|
Re: Arimaa rating deflation
« Reply #112 on: Apr 5th, 2006, 10:50pm » |
Quote Modify
|
on Apr 5th, 2006, 1:04pm, Fritzlein wrote:
If I were a programmer, I would automate this process somehow, but given my limited abilities I just kludge along the best I can. |
|
In my book, mathematician are a superset of programmers (it's just that they don't know it).
|
|
IP Logged |
|
|
|
Fritzlein
Forum Guru
Arimaa player #706
Gender:
Posts: 5928
|
|
Re: Arimaa rating deflation
« Reply #113 on: Aug 18th, 2006, 3:22pm » |
Quote Modify
|
on Sep 17th, 2004, 10:04am, 99of9 wrote:
Well now I've written a program to do some simulated annealing to determine the best fit ratings. Here's the output for that same dataset.
# Rating: 0.0 M
# Rating: -23.6 S
# Rating: 311.3 S+K-K
# Rating: 362.9 S+I
# Rating: 491.4 S+I-I
# Rating: 806.9 M+I-I
# Rating: 913.0 S+F-F
# Rating: 986.1 S+S-S
# Rating: 1007.1 M+F-F
# Rating: 1073.9 M+S-S
So I wasn't that far off by hand...
|
|
I just re-read this post from two years ago, and realized two things:
1) I doubt simulated annealing is necessary for the estimation of ratings, because I suspect there are no local minima other than the global minimum.
2) M+S-S is another name for ArimaaScoreP1. This is quite amusing, because after all the fuss about anchoring the system with a random mover rated zero, the un-anchored system is giving ArimaaScoreP1 a rating of 1178, i.e. roughly the rating of 1074 that 99of9 calculated it would have on an anchored scale!
One caveat is that ArimaaScoreP1 plays almost exclusively newcomers who haven't yet beaten ArimaaScoreP1, and if those opponents are on average overrated by about X points, then ArimaaScoreP1 is probably also overrated by about X points. I would guess that if the scale were not distorted by having all newcomers play the same bot first, then ArimaaScoreP1 would be rated below 800 on the current scale. In other words, our whole system would probably need to nudge up three hundred points to be consistent with a random player having a zero rating. It only looks like a match at the moment because ArimaaScoreP1 is overrated.
Even so, it is hilarious that the arbitrary choice of having newcomers enter at 1500 has anchored the ratings not far from the most intuitive non-arbitrary point of random=0. All that Omar has to do now to claim that our system is anchored, is to fix ArimaaScoreP1 to a rating of 1074!
(Of course, I still feel it is bad to use bots to anchor a rating system for reasons outlined earlier, but if anchoring the system merely means fixing ArimaaScoreP1 to a rating of 1074, it will make essentially no practical difference from the current system.)
|
|
IP Logged |
|
|
|
Fritzlein
Forum Guru
Arimaa player #706
Gender:
Posts: 5928
|
|
Re: Arimaa rating deflation
« Reply #114 on: Aug 19th, 2006, 10:38am » |
Quote Modify
|
I tried a new method of measuring rating inflation. I divided games into blocks of 1000, and then averaged the ratings of both players of all games played in each block. Thus if a high-rated player happened to play a bunch of games, that would pull the average up, and if a low-rated player played a bunch of games, that would pull the average down. Bots and humans both included.
As you can see from the graph above, that method is too volatile to get a good reading on inflation. So I revisited an earlier method, namely taking the average over players rather than over games. In the graph below, I averaged over the 100 most recent players to play a rated game. Bots and humans both included.
I attribute the dip in average rating shortly after 30,000 to an influx of new players at that time, and the beginning of the bot ladder, which made some of the previously-inactive low-rated bots into active players. That raises the question why the surge of new players from MetaFilter, around game 18500, didn't have a similar impact, but at that time the first bot for newcomers was ShallowBlue, playing unrated. Thus most of the MetaFilter newcomers never played any rated games.
My guess is that every surge of newcomers provides only a temporary drag on the average rating of active players, because if they leave soon their low rating drops out of the average while their donated points stay, but if they stay for a while, they usually get a higher rating.
From both of these graphs, we can see that the average player rating has risen from about 1500 to about 1600 over the lifetime of the server.
|
| « Last Edit: Aug 19th, 2006, 10:45am by Fritzlein » |
IP Logged |
|
|
|
OLTI
Forum Full Member
Arimaa player #1034
Gender:
Posts: 25
|
|
Re: Arimaa rating deflation
« Reply #115 on: Feb 9th, 2007, 3:24pm » |
Quote Modify
|
Check out the "p8 rating" on Top Rated Players
|
|
IP Logged |
|
|
|
Fritzlein
Forum Guru
Arimaa player #706
Gender:
Posts: 5928
|
|
Re: Arimaa rating deflation
« Reply #116 on: Feb 9th, 2007, 4:39pm » |
Quote Modify
|
I didn't post an updated graph, but I recently ran a query, and the average rating of the last 100 players to play a game has hovered just over 1600 for the last 10,000 games. I thought perhaps inflation was rampant, i.e. the trend might keep going up and up and up, but apparently it has leveled off a bit. For now the "average" player is rated somewhere around 1620.
By another measure, we could look at only the established players page, where you have to have RU 60 or less to be listed. The 61 established humans have an average rating of 1707, while the 48 established bots have an average rating of 1598. Combined, the average established player is rated 1659. This excludes newcomers, so it is a bit higher than my other figure of 1620.
|
| « Last Edit: Feb 9th, 2007, 6:10pm by Fritzlein » |
IP Logged |
|
|
|
omar
Forum Guru
Arimaa player #2
Gender:
Posts: 1003
|
|
Re: Arimaa rating deflation
« Reply #117 on: Jan 24th, 2008, 7:28pm » |
Quote Modify
|
Karl mentioned to me that he and some other players have been noticing continued inflation in the rating system. For a long time now we've discussed anchoring the rating system so that it does not drift. Additionally we wanted to anchor it so that the random bot has a rating of zero. Karl has mentioned earlier in this thread (Aug 18th, 2006) that perhaps the easiest way to do it would be to fix the rating of ArimaaScoreP1 to 1074 based on the work that Claude and Toby did to determine the rating of such a bot relative to the random bot. I guess my biggest hold back has been that I wanted to experiment and learn more about anchoring a rating system before taking the leap. Also I wanted to do it when we switched to a new rating system that was more resistant to abuse. But since Im never able to get around to working on that, I think I'll just do the easy thing now to reduce the drift.
I'll start by fixing the rating of ArimaaScoreP1 to 1074. After that we can perhaps also fix the rating of ArimaaScoreP2 if we are able to calculate its theoretical rating relative to P1.
Another thing we can do easily is change the value of the initial ratings that players start out at. Karl did some analysis and posted in another thread that the probability of a new player winning against ArimaaScoreP1 in their first and second game is 66.7% and 71.7%. Based on this we can probably guess a better value for a new player relative to what ArimaaScoreP1's fixed rating will be.
http://arimaa.com/arimaa/forum/cgi/YaBB.cgi?board=talk;action=display;nu m=1169996698
|
|
IP Logged |
|
|
|
clauchau
Forum Guru
bot Quantum Leapfrog's father
Gender:
Posts: 145
|
|
Re: Arimaa rating deflation
« Reply #118 on: Jan 25th, 2008, 7:09am » |
Quote Modify
|
Is ArimaaScoreP1 maximizing (the player's Arimaa score minus the opponent's Arimaa score) ? It would then not exactly be the same as the bot M+S-S used in Toby's and mine calcutations, which amounted to maximizing (the player's Arimaa score*1000000 minus the opponent's Arimaa score.
|
|
IP Logged |
|
|
|
|
Home
Help
Search
Members
Login
Register
Reply
Notify of replies
Send Topic
Print