Author |
Topic: Arimaa rating deflation (Read 30020 times) |
|
Fritzlein
Forum Guru
Arimaa player #706
Gender: 
Posts: 5928
|
 |
Re: Arimaa rating deflation
« Reply #90 on: Nov 2nd, 2004, 6:34pm » |
 Quote  Modify
|
on Nov 2nd, 2004, 4:53pm, 99of9 wrote:If you include bots in your definition of an "active player" what do you find?
|
|
If we include active bots the average rating has still gone up, albeit by only 80 points. So there is definitely inflation in the sense that the average rating of all active players has gone up, not just in the sense that humans are pulling ahead of the bots.
On the other hand, it is more difficult to separate the deflation of the rating of a consistent player from humans getting better at bashing bots. Since humans generally improve over time, we can't really test what happens over time to the rating of a human who doesn't get better or worse. But if we had such a human, I'll bet his rating would be lower now than a year ago.
|
|
 IP Logged |
|
|
|
RonWeasley
Forum Guru
Harry's friend (Arimaa player #441)
Gender:
Posts: 882
|
|
Re: Arimaa rating deflation
« Reply #91 on: Nov 3rd, 2004, 11:55am » |
Quote Modify
|
Here's something related but different.
If I don't play for a long time, I play worse in my first games after the layoff. I thought I noticed this when naveed and clauchau started playing again after periods of inactivity.
This makes me think a rating should decay as a function of inactive time. Maybe decay to 1500 with a time constant of 90 days. I shy away from recommending we actually do this since I worry about bad side effects.
Another penalty might be applied if a player always has the same opponent. This problem has been discussed before but not with a penalty in mind. Seems like a great opportunity for side effects here. This is why I would only duel with Crabbe.
|
|
IP Logged |
|
|
|
Fritzlein
Forum Guru
Arimaa player #706
Gender:
Posts: 5928
|
|
Re: Arimaa rating deflation
« Reply #92 on: Nov 3rd, 2004, 4:23pm » |
Quote Modify
|
In terms of accuracy of rating, it is reasonable to penalize people who have had a long layoff with no games. Probably the relative ratings at any given time would be slightly more accurate if we did that. However, rating systems usually shy away from such penalties on the grounds than they tend to deflate the system as a whole over time.
An alternative would be to increase the RU of anyone who hasn't played for a while. Typically you would expect players to get worse during a layoff, but they might get better. Perhaps I took time off to play a series against Bomb on my home computer, for example. Increasing the RU says, in effect, we don't know what happened during the layoff, but the longer the layoff, the less certain we are about the accuracy of the old rating.
I'd be game for Omar to implement this change to the current system independent of all the other changes we have discussed. Every week (or every day!) he could automatically add 1 to the RU of every account. Then players who are inactive for a long time get to start off at their old rating, but it will move fast when they rejoin, and in the mean time the high RU will reflect that their rating isn't currently accurate.
|
|
IP Logged |
|
|
|
omar
Forum Guru
Arimaa player #2
Gender:
Posts: 1003
|
|
Re: Arimaa rating deflation
« Reply #93 on: Nov 7th, 2004, 10:24pm » |
Quote Modify
|
Sounds like a good idea. I setup a cron deamon to increment the RU by 1 point (if it is less than 120) each week.
|
|
IP Logged |
|
|
|
Fritzlein
Forum Guru
Arimaa player #706
Gender:
Posts: 5928
|
|
Re: Arimaa rating deflation
« Reply #94 on: Nov 16th, 2004, 10:52am » |
Quote Modify
|
As it turns out, using Arimaazilla as a benchmark for inflation might not be a totally brilliant idea. Arimaazilla's rating apparently fluctuates a great deal depending on who its recent opponents have been.
There may be a general downward trend in Arimaazilla's rating, but there is enough noise that it isn't very conclusive
[Edit] Updated to January 4, 2005. Arimaazilla and Arimaanator are down, Bomb is up.
[Edit] Updated through January 25, 2005. Arimaanator has recovered somewhat to about 1650, but Arimaazilla is still under 1400, low by historical standards.
|
| « Last Edit: Feb 2nd, 2005, 5:58pm by Fritzlein » |
IP Logged |
|
|
|
fotland
Forum Guru
Arimaa player #211
Gender:
Posts: 216
|
|
Re: Arimaa rating deflation
« Reply #95 on: Nov 19th, 2004, 12:23am » |
Quote Modify
|
What a discouraging graph  I got bomb to 1800 in about 6 weeks, and its rating hasn;t improved since, with all the extra work I've put in.
Reality is that the rating system has deflated several hundred points as people have gotten better at the game. That early 1800 bomb would only be about 1600 now.
|
|
IP Logged |
|
|
|
omar
Forum Guru
Arimaa player #2
Gender:
Posts: 1003
|
|
Re: Arimaa rating deflation
« Reply #96 on: Nov 22nd, 2004, 11:21pm » |
Quote Modify
|
Wow nice. Looks like Im also taking a nose dive. I'd like to see Toby's graph
|
|
IP Logged |
|
|
|
Fritzlein
Forum Guru
Arimaa player #706
Gender:
Posts: 5928
|
|
Re: Arimaa rating deflation
« Reply #97 on: Mar 23rd, 2005, 9:24pm » |
Quote Modify
|
I updated the graph through mid-March, but since I changed the style I thought I would leave the old one up. Rather than having a dot per game, I have a dot per player per week, the weekly average of that player. Also I replace Omar with Naveed, since Naveed has been the most active player over the life of the server.
We can see that Arimaazilla is slightly low by historical standards, while Arimaanator is quite low. There are too many active Arimaanator-beaters out there nowadays to let Arimaanator get any traction from new players.
Bomb is not low by historical standards, but is still stuck hovering just over 1800, which must be frustrating given that it has only played when under active development. The gap between Bomb and the challenge prize does not seem to be narrowing even though Bomb is getting better.
|
|
IP Logged |
|
|
|
Fritzlein
Forum Guru
Arimaa player #706
Gender:
Posts: 5928
|
|
Re: Arimaa rating deflation
« Reply #98 on: Mar 24th, 2005, 2:34pm » |
Quote Modify
|
on Mar 24th, 2005, 12:00pm, Arimanator wrote:| I rate a measly 1100 and I just beat bot arimaazon twice in a row... |
|
You are an excellent example of how new players can contribute to either inflation or deflation. At first, as your rating dropped from 1500 to 1100, you contributed about 100 points each to the ratings of Arimaalon and Arimaazilla. If you leave now and never play again, you will have injected points into the rating system, contributing to inflation.
On the other hand, you are now underrated, and your RU is down to 30, so the system considers that you are an established player. Now if you beat up on the bots until your rating is back up to 1500, you will take points from them exactly equal to the number of points you gain, i.e. you will draw 400 rating points back out when you only put 200 in, even if you start and end your career at the same rating of 1500.
One reason the bots are low at the moment is that Ryan Cable came through before you, lost many points to the bots at a high RU, then won them back at a low RU. The net effect was pushing down the bots ratings. In fact he has now pushed himself up to 1700, so he drew even more points out of the system beyond what he got for getting back up to 1500.
On the other hand, this is somewhat balanced by people who only lose a few games and don't come back, and therefore each donate a few rating points into the pool.
|
|
IP Logged |
|
|
|
Fritzlein
Forum Guru
Arimaa player #706
Gender:
Posts: 5928
|
|
Re: Arimaa rating deflation
« Reply #99 on: Mar 24th, 2005, 3:24pm » |
Quote Modify
|
While I was back on the subject, I couldn't resist another picture. This graph shows the week-by-week average rating of all Arimaa players who played at least one rated game during that week. Also there is a line for human players only.
The average active human player seems to be a bit stronger than the average active bot most of the time, but not always, and it wasn't that way at first. Also, although everyone starts at 1500, the average active player is substantially higher.
This graph is too noisy for me to tell whether it is rising or not: At least we aren't obviously experiencing an increase in average rating at the moment (which would be a type of inflation), whereas I think we are quite probably experiencing the type of deflation whereby 1800 designates a stronger player than it used to.
|
|
IP Logged |
|
|
|
Fritzlein
Forum Guru
Arimaa player #706
Gender:
Posts: 5928
|
|
Re: Arimaa rating deflation
« Reply #100 on: Jul 24th, 2005, 6:57pm » |
Quote Modify
|
Now that newcomers have more bots to play, more rungs in the ladder, so to speak, it seemed appropriate to update my graph of bot ratings to include them.
I thought that having a greater spectrum of bots might mean that the rating of each would swing less wildly, but it appears not to have happened. Arimaalon, which is now the first rated bot for everyone, has perhaps taken on some of the uncertainty that used to go to Arimaazilla, but the latter still has regular ups and downs. Arimaazon should by rights be consistently higher than Arimazilla, but there isn't much separation, which may be a result of some people preferring to play fast.
The rating of the bots still seems hugely dependent on who is playing them, and how persistent they are. Blue22 has beaten Arimaanator down under 1600 for a while, which is historically very low, and which has allowed Arimaazon to cross over and temporarily have a higher rating, even though there is little question in my mind that Arimaanator is the stronger bot.
Maybe once the Arimaa playing population grows to the point that the influx of new players is more steady, the bot ratings will be more predictable. For now, however, the ratings of the standard bots in the lobby seem to have more to do with whether the latest newbie likes to move up to new challenges as soon as possible, or likes to stay with one bot until gaining complete mastery before moving on.
I wonder whether a better series of steps would be to have all lobby bots playing at 2 minutes per move, but moving quickly to internal limits so the folks can blitz along if they so choose. Perhaps the championship P1 and P2 bots would be good for this purpose. Another purpose would be served by mixing things up: at present the first four bots all use the same evaluation function. Newcomers might learn more from, say, ShallowBlue, LocP1, Arimaazilla, CluelessP1, GnobotP2, BombP2 than from the current setup. Just a thought.
|
|
IP Logged |
|
|
|
omar
Forum Guru
Arimaa player #2
Gender:
Posts: 1003
|
|
Re: Arimaa rating deflation
« Reply #101 on: Jul 27th, 2005, 6:58am » |
Quote Modify
|
I like the idea of using the fixed performance CC bots in the lobby since they have such different styles of play.
|
|
IP Logged |
|
|
|
Fritzlein
Forum Guru
Arimaa player #706
Gender:
Posts: 5928
|
|
Re: Arimaa rating deflation
« Reply #102 on: Sep 27th, 2005, 11:34pm » |
Quote Modify
|
My previous graph of the average rating of players by week was too choppy to convey much. I have generated a new graph of the monthly average rating of all active human players. That is to say, I divided the game database into 30-day chunks; within each chunk I calculated the average rating of each human who played at least one rated game; then I averaged the ratings of all humans active in that chunk.
It is clear that we are experiencing a gradual rating inflation. The average human rating, which started around 1500, has climbed to around 1600.
I'm guessing that if the system were anchored with fixed-performance bots of fixed rating, we would have seen rating inflation of more than 300 points in the same time span. As it is, the system seems to be steering a resonable middle course between fixing the average rating of the playing pool (which is going up) and fixing the rating of an unchanging skill level (which is going down).
|
| « Last Edit: Sep 27th, 2005, 11:36pm by Fritzlein » |
IP Logged |
|
|
|
Ryan_Cable
Forum Guru
Arimaa player #951
Gender:
Posts: 138
|
|
Re: Arimaa rating deflation
« Reply #103 on: Jan 29th, 2006, 8:46pm » |
Quote Modify
|
I just noticed that it is possible to sort the New Players list by ratings. (It seems to list humans and bots that joined within a year ago.) There are 159 new players rated <1500, with a mean rating of 1381, collectively loosing 18892 points. There are 69 new players rated >1500, with a mean rating of 1598, collectively gaining 8244 points.
Combined the new players lost 10648 points. Most of these players played a few games at RU~=120 against opponents with RU~=30 and then left, so they added ~2662 points to the pool of active players. There are 42 players with RU<=50, and 70 players with RU<=80, so in the last year, we added between ~38 and ~63 points per active player.
Only 2 new players are rated >=2000, blue22 and the inactive Arimanator. Only 1 other new human is rated >=1700, the infrequently active OLTI. Only 8 other new humans are rated >=1600; 5 of whom have been seen within 31 days, and 2 of whom have RU<=80. I can only hope this doesn’t mean there is a drop off in the rate of potential 2000+ players joining.
|
|
IP Logged |
|
|
|
Fritzlein
Forum Guru
Arimaa player #706
Gender:
Posts: 5928
|
|
Re: Arimaa rating deflation
« Reply #104 on: Jan 31st, 2006, 10:23am » |
Quote Modify
|
Nice analysis as usual, Ryan. You inspired me to make another graph. I decided to base this one on game number rather than on date. My definition of "Average Rating" of the Arimaa population was to take the last rating of each of the last 100 players to have played a rated game. I included both bots and humans. At present these 100 players include everyone who has played a rated game within approximately the past two months, but earlier (when the playing pool was smaller) it probably covered a longer time span.
The point I wanted to make was that newbies are a source of rating deflation, at least temporarily. When someone joins and drops 240 points to Arimaalon who gains 60 points, the system loses a net 180 points. It isn't until that newbie stops playing and drops off the end of the list of active players that we can say the system has gained a net 60 points.
Thus we see clearly at the beginning, when the ratings still included all active players, there was a clear deflationary trend. It wasn't until people started cycling out of the playing pool that inflation kicked in, inflation based on the fact that most players who become inactive have ratings below 1500.
Despite approximately 70 points of rating inflation, I'm guessing a player rated 2000 today is 100 to 200 points stronger than a player rated 2000 two years ago. I'm curious how it would affect the rating inflation trend if the average active player weren't getting significantly stronger. Perhaps if the deflationary influence of everyone learning together wasn't so strong, then the inflation would really take off. Or maybe the inflation isn't so much affect by that. I dunno.
|
|
IP Logged |
|
|
|
|
Home
Help
Search
Members
Login
Register
Reply
Notify of replies
Send Topic
Print