Author |
Topic: Arimaa rating deflation (Read 30066 times) |
|
Fritzlein
Forum Guru
Arimaa player #706
Gender: 
Posts: 5928
|
 |
Re: Arimaa rating deflation
« Reply #135 on: May 21st, 2008, 8:42am » |
 Quote  Modify
|
Omar, why not just anchor ArimaaScoreP1 to a rating of 1000? That will counter the immediate problem of a gradual upward drift of ratings. (I'm coming to believe that we are experiencing rating increases that are greater than our collective improvement in skill.) Our system would then be dual-anchored by starting beginners are 1500 and fixing ArimaaScoreP1 to 1000.
The theoretical problem of how ArimaaScoreP1 should be rated relative to a random mover is less important, partly because accuracy is meaningless. Because the ratings are not transitive, there is no absolute scale, so an approximate answer is good enough. It doesn't make sense to wait to anchor until we have a accurate answer to an unanswerable question. We should rather take steps to fix an observed, practical problem.
Anchoring one bot should have an impact in the right direction. Is it enough? We could wait and see what the effect is. I tend to believe that anchoring the whole continuum of fixed performance bots would be a medicine worse than the disease. I expect the best practical outcome from anchoring some small number of bots at the bottom of the ladder. But we won't figure out what is best by theorizing. Let's start trying stuff!
|
| « Last Edit: May 21st, 2008, 8:50am by Fritzlein » |
 IP Logged |
|
|
|
Adanac
Forum Guru
Arimaa player #892
Gender:
Posts: 635
|
|
Re: Arimaa rating deflation
« Reply #137 on: Jun 3rd, 2008, 7:19am » |
Quote Modify
|
on Jun 3rd, 2008, 5:18am, omar wrote:
Omar, thanks for addressing this issue - this should put a lid on the inflation problem. I've always liked the idea of decreasing the initial rating for new players.
|
|
IP Logged |
|
|
|
Fritzlein
Forum Guru
Arimaa player #706
Gender:
Posts: 5928
|
|
Re: Arimaa rating deflation
« Reply #138 on: Jun 8th, 2008, 9:20am » |
Quote Modify
|
on Jun 3rd, 2008, 5:18am, omar wrote:| I think I am just waiting to decide what to do about the initial rating of new players. If we want to change that as well then I would like to make both changes at once. |
|
Making both changes at once sounds fine. It may be an over-correction that brings about deflation, but it's better to try something than not try anything.
Quote:
Those stats show that newcomers win about 2/3 against ArimaaScoreP1 in their very first game, which would suggest that they are 200 rating points better than ArimaaScoreP1, even when they have no experience. Therefore it would make more sense to fix ArimaaScoreP1 to 1000 and set initial rating of new players to 1200. Also, however, the stats show that players improve approximately 100 points between their first game and second game. Therefore we should either give a 100-point bonus for completing a game, or we should set the initial rating to 1300, which amounts to basically the same thing given that the first bot everyone plays will have a fixed rating, so we aren't worried about folks who only play once injecting points into the system.
|
| « Last Edit: Jun 8th, 2008, 9:25am by Fritzlein » |
IP Logged |
|
|
|
omar
Forum Guru
Arimaa player #2
Gender:
Posts: 1003
|
|
Re: Arimaa rating deflation
« Reply #139 on: Jun 11th, 2008, 2:20pm » |
Quote Modify
|
Sounds good I will fix the rating of ArimaaScoreP1 to 1000 and start new players at 1300.
Karl, can you make a graph of the rating history of ArimaaScoreP1 and maybe some of the other fixed performance bots. This will show clearly that there is inflation. I was looking at the numbers on the screen and there definitely is a slow and gradual rise in the rating of the fixed performance bots. Having some graphs to easily see this and compare it in the future would be nice.
|
|
IP Logged |
|
|
|
Fritzlein
Forum Guru
Arimaa player #706
Gender:
Posts: 5928
|
|
Re: Arimaa rating deflation
« Reply #140 on: Jun 11th, 2008, 7:00pm » |
Quote Modify
|
I guess I should have posted the stats here instead of in the WHR thread, but I can at least link it:
http://arimaa.com/arimaa/forum/cgi/YaBB.cgi?board=talk;action=display;nu m=1207699394;start=15#15
on Jun 11th, 2008, 2:20pm, omar wrote:| Sounds good I will fix the rating of ArimaaScoreP1 to 1000 and start new players at 1300. |
|
Thank you! I will watch the results with interest. I expect this will not only stabilize ratings, but noticeably deflate them from current levels, which isn't all bad if it is correcting for past inflation.
|
| « Last Edit: Jun 11th, 2008, 7:05pm by Fritzlein » |
IP Logged |
|
|
|
aaaa
Forum Guru
Arimaa player #958
Posts: 768
|
|
Re: Arimaa rating deflation
« Reply #141 on: Jun 15th, 2008, 2:55pm » |
Quote Modify
|
Wouldn't it make more sense to anchor the aggregate of the ratings of all the bots on the ladder?
|
| « Last Edit: Jun 15th, 2008, 2:59pm by aaaa » |
IP Logged |
|
|
|
Fritzlein
Forum Guru
Arimaa player #706
Gender:
Posts: 5928
|
|
Re: Arimaa rating deflation
« Reply #142 on: Jun 15th, 2008, 4:49pm » |
Quote Modify
|
on Jun 15th, 2008, 2:55pm, aaaa wrote:| Wouldn't it make more sense to anchor the aggregate of the ratings of all the bots on the ladder? |
|
What does it mean to anchor the aggregate? Do you mean that the ratings of the bots can move relative to each other, but the average rating is held constant? For example, if a newcomer enters and gives ArimaaScoreP1 30 points, then the other 15 bots we are anchoring on as an aggregate each lose 2 points to compensate?
That's an interesting idea to prevent points from entering or leaving the anchor pool. The anchor bots could still get out of synch with the non-anchor players, if most of the games didn't involve an anchor player, but the larger anchor would be better on that score than an anchor of one (ArimaaScoreP1). The disadvantages I see right off would be:
1. Slight implementation hassle: ratings couldn't be stored as integers any more or rounding error would defeat the mechanism.
2. Initialization: How would one decide on the total rating points of the anchor pool?
3. Helping bot-bashers: Someone sucking points from a particular bot reaches a limit until someone else gives that bot points, but now points could be given to the bashed bot indirectly.
My own attitude is that I have long since given up on having accurate ratings in any system that includes (A) bots and (B) self-selection of opponents. It's impossible. However, if we are forgetting accuracy and just thinking about what the best inflation anchor would be, I like an anchor pool with sloshing around ratings better than I like anchoring the rating of each fixed-performance bot. Whether I like an anchor pool better than tinkering with the ratings of newcomers depends on how well the latter works. If we can get a non-inflationary and non-deflationary system by just changing one number, then it's a much easier way to address the whole issue.
|
|
IP Logged |
|
|
|
Fritzlein
Forum Guru
Arimaa player #706
Gender:
Posts: 5928
|
|
Re: Arimaa rating deflation
« Reply #143 on: Jul 20th, 2008, 8:44am » |
Quote Modify
|
on Jun 11th, 2008, 2:20pm, omar wrote:| Sounds good I will fix the rating of ArimaaScoreP1 to 1000 and start new players at 1300. |
|
It has been almost six weeks since the anti-inflation changes went into effect, so I thought I would check in on what effect they were having. I took all the players who had accounts prior to the new rules, and calculated the net change in their ratings due to playing newcomers and ArimaaScoreP1. As a group the old-timers (bots and humans both) lost 758 rating points. So if deflation is what we wanted, deflation is what we got. Presumably the system will continue to hemorrhage points until it finds a new equilibrium at a much lower point.
However, if we scratch beneath the surface a bit, that deflationary number seems less like a trend and more like a quirk. ArifSyed purposely deflated his own rating by 455 points over 16 losses to ArimaaScoreP1. We worried a bit that fixing the rating of a bot would allow people to arbitrarily inflate their ratings by playing it over and over, but we didn't consider that it also makes the perfect sinkhole for sandbaggers. In the same time period, ArifSyed has increased his rating by 122 points by beating newcomers, many more points than he would have taken if his rating had been accurate.
In a sense, ArifSyed is acting like a deflation pump himself, by taking rating points from newcomers and making them disappear into thin air. 333 points of the net deflation was his direct doing.
Still, that leaves 425 points of deflation that is a "natural" result of having ArimaaScoreP1 fixed at 1000, and having newcomers enter at 1300. It's still too early to tell, but I think we may have over-corrected for the inflation we were previously experiencing.
|
| « Last Edit: Aug 5th, 2008, 8:52am by Fritzlein » |
IP Logged |
|
|
|
aaaa
Forum Guru
Arimaa player #958
Posts: 768
|
|
Re: Arimaa rating deflation
« Reply #144 on: Jul 20th, 2008, 9:09am » |
Quote Modify
|
I still say that there should be two ratings: one for human-bot games and one for homogeneous (bot-bot/human-human) ones.
|
|
IP Logged |
|
|
|
mistre
Forum Guru
Gender:
Posts: 553
|
|
Re: Arimaa rating deflation
« Reply #145 on: Jul 20th, 2008, 9:49am » |
Quote Modify
|
on Jul 20th, 2008, 8:44am, Fritzlein wrote:
In a sense, ArifSyed is acting a deflation pump himself, by taking rating points from newcomers and making them disappear into thin air. 333 points of the net deflation was his direct doing.
|
|
Hopefully he is only making the rating points disappear into thin air and not the newcomers themselves. 
|
|
IP Logged |
|
|
|
Fritzlein
Forum Guru
Arimaa player #706
Gender:
Posts: 5928
|
|
Re: Arimaa rating deflation
« Reply #146 on: Jul 20th, 2008, 10:52am » |
Quote Modify
|
on Jul 20th, 2008, 9:49am, mistre wrote:| Hopefully he is only making the rating points disappear into thin air and not the newcomers themselves. |
|
Heh. I hope my editor catches all the unclear antecedents and suchlike before my book hits the presses. 
|
|
IP Logged |
|
|
|
Arimabuff
Forum Guru
Arimaa player #2764
Gender:
Posts: 589
|
|
Re: Arimaa rating deflation
« Reply #147 on: Jul 20th, 2008, 12:02pm » |
Quote Modify
|
on Jul 20th, 2008, 10:52am, Fritzlein wrote:
Heh. I hope my editor catches all the unclear antecedents and suchlike before my book hits the presses. |
|
Don't worry Honoré de Balzac one of our geniuses of literature was known for his many bloopers.
He once wrote: "Oh oh, he said in Portuguese."
How do you say "oh, oh" in Portuguese? 
|
|
IP Logged |
|
|
|
aaaa
Forum Guru
Arimaa player #958
Posts: 768
|
|
Re: Arimaa rating deflation
« Reply #148 on: Aug 4th, 2008, 8:29pm » |
Quote Modify
|
Interesting how David Fotland's suggestion concerning the ratings ended up being followed more than four and a half years later.
|
|
IP Logged |
|
|
|
Fritzlein
Forum Guru
Arimaa player #706
Gender:
Posts: 5928
|
|
Re: Arimaa rating deflation
« Reply #149 on: Aug 5th, 2008, 8:50am » |
Quote Modify
|
Wow, that is ironic. Years later we implemented what he suggested in the first post. But the weirdest thing is that Fotland wanted to lower the ratings of new players because of deflation, and we finally decided to do it because of inflation. Go figure.
|
|
IP Logged |
|
|
|
|
Home
Help
Search
Members
Login
Register
Reply
Notify of replies
Send Topic
Print