Author |
Topic: Unrating Games / Integrity of Ratings (Read 4242 times) |
|
Fritzlein
Forum Guru
Arimaa player #706
Gender: 
Posts: 5928
|
 |
Unrating Games / Integrity of Ratings
« on: May 25th, 2006, 10:29am » |
 Quote  Modify
|
I remember the extreme frustration I felt when I was working through the bots of my day. When I lost games due to interface issues, exacerbated by my slow connection, it seemed so unfair I contemplated quitting Arimaa entirely. It is ridiculous for the ratings to reflect the speed and stability of one's connection to the server rather than one's ability to play Arimaa.
The solution, of course, is a more stable interface. Until a better interface could be developed, however, Omar offered the band-aid of unrating games lost due to interface and connection issues. In theory this could make the ratings more accurate, because players wouldn't lose rating points due to techinical issues on games they were otherwise winning.
Unfortunately, it has become common practice for players to unrate games they were otherwise losing. Even if the direct cause of a player losing the game is a disconnection or interface problem, to unrate a lost game that would probably be lost anyway makes the ratings less accurate, not more. Moreover, it appears probable that, in at least some cases, players have intentionally lost on time in positions that have become uncomfortable, merely to preserve the ability to make the game unrated. In short, the presence of Omar's band-aid is probably making the problem worse, rather than better.
I don't want to make a big fuss about ratings distortions due to retrospective unrating of games, because there are two problems that are much more serious. The first, as I've already mentioned, is the unstable interface. Any solution that doesn't improve the interface is bound to be problematic in some way.
The second problem is that ratings are going to be inaccurate anyway as long as the majority of games are human vs. bot games. It is common for players to inflate their ratings by incessantly playing bots they know how to beat. Also, since the introduction of the bot ladder, it has become common for players to deflate their ratings by incessantly playing bots they can't beat. Human vs. bot games demonstrate starkly how non-transitive the ratings are.
I've come to a point where I hardly care about human vs. bot games any more. Thirty-eight of my last fifty games have been against human opponents; the twelve bot games were mostly finishing up the bot ladder plus some stabs at bot-bashing that didn't hold my interest, so my percentage of games against bots may go down even further in the future. In this frame of mind, I'm much less concerned about the integrity of the ratings than I used to be, because the ratings are 90% a reflection of games versus bots. A year ago I inflated my rating to a record high by playing bots; now I am a better player (I hope!) but lower-rated. Who cares?
People who do care about ratings, however, should help Omar generate solutions to this problem. I can think of a few to get the discussion started:
1) Get rid of the feature for unrating games lost on time.
2) Keep the feature, but if anyone unrates a game they were losing, ban them from using the feature any more.
3) Instead of allowing one's most recent time loss to be unrated, allow it to be resumed from the final position. That would prevent people from avoiding uncomfortable positions. True, it would still allow people to gain extra thinking time in difficult positions, but this is a smaller issue that being able to dodge a lost position entirely. Also the player who won on time could be automatically compensated with (say) an extra three mintues of thinking time in reserve when the game resumes.
4) Put a timestamp in the interface, just as they have on the Internet Chess Club, so as to distinguish Internet lag from thinking for too long.
5) Just leave everything as is, accepting ratings distortions as a fact of life, but add a new category of rating, the "real" rating, which is based solely on human vs. human games. There is a greater chance of ultimately preserving the accuracy of a pure human vs. human rating, because there is more transitivity in human vs. human games, and because a percentage of the human vs. human games come from tournaments where the opponents are assigned, rather than self-selected.
I'm curious what everyone else thinks, but particularly curious what Omar is inclined to do about the issue, if anything.
|
|
 IP Logged |
|
|
|
RonWeasley
Forum Guru
Harry's friend (Arimaa player #441)
Gender:
Posts: 882
|
|
Re: Unrating Games / Integrity of Ratings
« Reply #1 on: May 25th, 2006, 12:28pm » |
Quote Modify
|
Indeed, this is a topic people should comment on, but I didn't want to be the only one and seem like a ratings prefect. Thank you, Fritzlein, for taking the initiative.
I agree it is a Slytherin cheat to unrate a game lost due to play and not connection. Usually it is impossible to tell the difference, but now we have cases where players are admitting to it. It's weird, but I feel better about a person being honest, even about cheating. I had to stop and ask myself how I could feel this way.
The answer for me is that I would be more worried if the ratings had more integrity otherwise. However, anybody can get any rating they want through rated bot bashing. So if somebody manipulates their rating through timeouts, it's the same effect, just less honorable. If they admit to it, I know their rating isn't accurate, but for me it stops there. There is no increased risk of anybody cheating in muggle-muggle games because it can't be done.
One adverse effect is tournament seeding. This would be more serious if more money was on the line, but for me it's not enough to worry about. Again, if you want to manipulate your tournament seed you can do rated bot bashing (or sand bagging).
My inclination is to leave things as they are and allow people who want accurate ratings to behave honorably. People who break the honor system can enjoy a higher rating, but the quality of your play speaks for itself. This works both ways. For example, most of us know jdb plays much better than his rating. It's low, I think, because he prefers to play only hard opponents. This is an example of an inaccurate rating for a completely legitimate reason. Our ratings aren't fooling any of the experienced players.
Another point of view, however. is how arimaa looks to the outside world. If we get a reputation for cheating, say goodbye to any prospect of corporate sponsorship. Consider the value of your rating if everyone automatically assumes you cheated to get it. Nobody would take any of us seriously. Why would anybody new want to play? Hmm. I may have changed my mind.
|
|
IP Logged |
|
|
|
jdb
Forum Guru
Arimaa player #214
Gender:
Posts: 682
|
|
Re: Unrating Games / Integrity of Ratings
« Reply #2 on: May 25th, 2006, 1:02pm » |
Quote Modify
|
Personally, I don't worry about the ratings. Its fairly easy to tell how strong a person is by looking at a few of their recent games anyway.
|
|
IP Logged |
|
|
|
Janzert
Forum Guru
Arimaa player #247
Gender:
Posts: 1016
|
|
Re: Unrating Games / Integrity of Ratings
« Reply #3 on: May 25th, 2006, 1:26pm » |
Quote Modify
|
I don't like the idea of people using this for reasons other than interface or server problems. Also some of the reasons given that I had seen recently did leave me feeling rather uncomfortable with them (and no don't even think I'm talking about you Arimanator none of these were your games I saw).
But at the same I'm not sure there is a practical fair way to handle it other than just living with it as it is now or shutting it off completely. I have a feeling at this stage shutting it off is probably going to be more detrimental overall than just living with it. At least until the interface/server is a little more stable for most players.
One thing that I think would help lessen the impact of skewed overall ratings would be seeing split out hvb, hvh and possibly seperatad time control bracket ratings.
On a side note, I do like 99of9's idea of only having the bot play quality games when rated and should I ever get a bot going will probably do the same.
Janzert
|
|
IP Logged |
|
|
|
nbarriga
Forum Guru
Almost retired Bot Developer
Gender:
Posts: 119
|
|
Re: Unrating Games / Integrity of Ratings
« Reply #4 on: May 25th, 2006, 2:19pm » |
Quote Modify
|
I like the idea of different ratings, HvH, HvB and BvB, and maybe 2 or 3 different ratings based on time controls.
Also i strongly support the idea of only some games being rated, like tournament games, but that would require more frequent tournaments, maybe 4 times a year. All the tournaments need not be equal, you can have the postal one, the WC, a fast and a blitz one, and maybe some other ones.
|
|
IP Logged |
|
|
|
chessandgo
Forum Guru
Arimaa player #1889
Gender:
Posts: 1244
|
|
Re: Unrating Games / Integrity of Ratings
« Reply #5 on: May 25th, 2006, 2:31pm » |
Quote Modify
|
As jdb says, never mind the rating, at least as long as it doesn't come into account for the Worldchampionship and other tournaments' entrance, as it might be the case in the future when (if ?) the community grows to thousands of regular players.
As for Arimanator's complaints about PoTM, I think Omar made his point quite clear, and fully understandable : it is designed to promote h vs h games, and especially for newcomers who essentially play games vs bots through the ladder.
I have to say that at arimaa as well as at any other strategy game, playing stronger people and asking them advice is the better way to make progress. So the concept that a win on a beginner is as valuable as any other win is a good one to make the whole cummunity grow stronger by making a game with a lesser player attractive for the stronger as well.
In the last weeks a lot of stronger guys agreed to play with me and even challenged me sometimes, Nathan first. I owe them a lot.
So if your insults are directed at me, I just despise them, and if they're directed at Nathan, I lack words to tell you what I think of a guy who denies the great work accomplished by him.
As for 99's Gnobot playing well only in rated games, I don't understand the point, but he makes what he wants with it, had he not done it, we wouldn't be able to botbash it neither.
Jean
|
|
IP Logged |
|
|
|
chessandgo
Forum Guru
Arimaa player #1889
Gender:
Posts: 1244
|
|
Re: Unrating Games / Integrity of Ratings
« Reply #6 on: May 25th, 2006, 4:00pm » |
Quote Modify
|
I'm beginning to lack english words so I'd rather go on in french to be sure to stay within the limits of decency.
Sombre crétin ! Tu devrais envisager de garder tes conneries sur le PoTM, ce qui me permettrais de garder les miennes sur le botbashing. Pour ce qui est des insultes dont tu as été victime, aucune ne venait ni de moi ni de Nathan.
Ceci étant dit tu peux me dénigrer autant que tu veux, les insultes hors de l'échiquier étant tout à fait méprisables, surtout venant de quelqu'un qui n'a jamais joué avec moi. Que tu te permettes d'insulter Nathan par contre me met hors de moi.
Jean
|
|
IP Logged |
|
|
|
PMertens
Forum Guru
Arimaa player #692
Gender:
Posts: 437
|
|
Re: Unrating Games / Integrity of Ratings
« Reply #7 on: May 25th, 2006, 4:45pm » |
Quote Modify
|
#1 without a solution is unfair to those who think that rating is important
I dislike #2 because I prefer Omars way of NOT checking into games and just hoping that nobody is abusing this
In addition it can be really difficult to judge positions.
My favourite solution would be #3: the resume function.
I dislike #4 because "higher reasons" can be more than just disrupted connections ... why are phone calls (emergency or not) treated different ?
or a broken keyboard ?
I like #5 ... a hvh rating and a hvb rating are two very different informations (and I like additional infos)
|
| « Last Edit: May 25th, 2006, 8:39pm by PMertens » |
IP Logged |
|
|
|
unic
Forum Guru
Arimaa player #1878
Gender:
Posts: 63
|
|
Re: Unrating Games / Integrity of Ratings
« Reply #8 on: May 25th, 2006, 5:25pm » |
Quote Modify
|
on May 25th, 2006, 10:29am, Fritzlein wrote:
1) Get rid of the feature for unrating games lost on time.
|
|
I would definitely vote for this.
I also vote for not allowing players to force unrated games upon bots that don't want them.
If you want to fool around with silly handicaps, be ready to accept that your rating might take a hit.
If you care that much about your rating (... which I don't understand why anyone would ...) - then don't fool around. Or play against people or bots who are happy to play unrated games, and don't mind if you leave the game to time out or resign when it is barely off the ground.
It is very annoying for me, and a waste of time, to see Fairy play the start of an interesting game, and then the opponent leaves the game, and in fifteen minutes it times out, or the opponent resigns while there is still plenty of play left. Not having the possibility of forcing an unrated game would discourage such wasteful behaviour.
Fairy is not going to be available on the server for people to play against anymore, until I have found a way around this issue of people forcing unrated games on it... I wouldn't mind the games being unrated if people took them seriously - but now, it seems too easy to not take them seriously. Having all games rated would discourage such foolish behaviour.
Arimanator in particular did this several times, but others have done so as well. Therefore, Fairy is no longer available. It's a shame... I am grateful to those humans who played proper games until the end against it, whether they lost or won. I learned much about Fairy's weaknesses and strength from those games.
Finally, I would like to quote the section on sportsmanship, which is linked to from the gameroom:
Quote:
Try to avoid resigning
Although it is possible to resign a game of Arimaa, you should only use this option for practice or non-rated games. When you enter a serious game of Arimaa, think of it as a duel to the finish and never give up. Do not resign even if it looks like you are going to lose the game. Play it to the end. There are many examples of games where a player has actually won even though they were short on material. If you can pull off such a victory, it's even more exciting to watch.
|
|
And even if you can't pull of a victory, surely a good struggle is better than throwing the towel in at the slightest sign of a problem.
(Okay, I'll admit that I am far from perfect - I've resigned games against bots, especially when I've been somewhat in a hurry... but in theory, I do agree with what the above quote says.)
So, if you mean to play a serious game, don't resign. If you don't mean to play a serious game, don't play my bot and waste its and my time!
|
|
IP Logged |
|
|
|
unic
Forum Guru
Arimaa player #1878
Gender:
Posts: 63
|
|
Re: Unrating Games / Integrity of Ratings
« Reply #9 on: May 25th, 2006, 6:28pm » |
Quote Modify
|
on May 25th, 2006, 6:23pm, Arimanator wrote:
I won't frequent a place where some of the people wear a symbolic yellow star. |
|
I wear a black Miscatonic University T-shirt, not a yellow star. ^^
|
|
IP Logged |
|
|
|
unic
Forum Guru
Arimaa player #1878
Gender:
Posts: 63
|
|
Re: Unrating Games / Integrity of Ratings
« Reply #10 on: May 25th, 2006, 6:49pm » |
Quote Modify
|
(Who's not very bright and not recognizing sarcasm?  )
Well, chess servers and other game servers allow both formulas and ignore lists - it's a common and useful feature. See for example Internet Chess Club, Free Internet Chess Server, ChessBase's Playchess.com and the UoA's Generic Game Server, which all have some variation or other of this.
I don't see the problem with that. In fact, if Arimaa is to continue growing, something like formulas and being able to ignore certain people/bots would be very useful sooner or later. 
|
|
IP Logged |
|
|
|
unic
Forum Guru
Arimaa player #1878
Gender:
Posts: 63
|
|
Re: Unrating Games / Integrity of Ratings
« Reply #11 on: May 25th, 2006, 7:07pm » |
Quote Modify
|
(Apart from silly comments - I guess trying to keep the tone light didn't work...) As I pointed out, this is a common feature in many other game servers. I do not see what is wrong with it.
|
|
IP Logged |
|
|
|
chessandgo
Forum Guru
Arimaa player #1889
Gender:
Posts: 1244
|
|
Re: Unrating Games / Integrity of Ratings
« Reply #12 on: May 25th, 2006, 7:41pm » |
Quote Modify
|
Hum, I put back my clown hat that I shouldn't have put aside  to remind you guys that facism and the Shoah and not funny matters and people might be offended seeing us speaking so lightly of it ... So I want hereby to apologize to all the fervent nazis for the lightness of what is being said here !
|
|
IP Logged |
|
|
|
unic
Forum Guru
Arimaa player #1878
Gender:
Posts: 63
|
|
Re: Unrating Games / Integrity of Ratings
« Reply #13 on: May 25th, 2006, 7:46pm » |
Quote Modify
|
[ this part of the post was deleted since it was a response to a post by Arimanator; and that post was also deleted ]
And as far as the annual challenge goes, I am not concerned with that, and was not planning to take part in it (for various reasons... mainly, I can't be bothered getting Fairy to run on Linux).
What I do plan to do is code a graphical interface to Fairy and release it, once I am happy with it. Then people can download it and play with it if they want to. Until then, I would prefer to have it play only games that are useful for me in learning more about Fairy and Arimaa - and if that means sticking to playing other bots, that is a sad state of things.
(And while we're at it - I had planned to wait with bringing this up until I actually started programming it - but what would people like to see in a graphical interface? I have downloaded the game databases, and was hoping to make them accessible in some face... compare the way the Othello program WZebra gives access to the Thor game databases. What other features would be viewed as useful?)
|
| « Last Edit: May 25th, 2006, 8:19pm by Forum Admin » |
IP Logged |
|
|
|
unic
Forum Guru
Arimaa player #1878
Gender:
Posts: 63
|
|
Re: Unrating Games / Integrity of Ratings
« Reply #14 on: May 25th, 2006, 7:54pm » |
Quote Modify
|
[ this part of the post was deleted ]
Omar, Fritzlein, Swynndla, Arimanator, everybody else? Should I remove *my* bot? Or should I keep it here? I will abide by what the majority say (I will count this by bedtime tomorrow night...). If people would rather not have Fairy here, there are plenty of other abstracts I would be curious to try programming an AI for... the only reason Arimaa was the one I actually chose was the access to easy testing via the server.
This is my final post on this topic. I still think accept/decline formulas and ignore lists (which are a standard function on many other game servers) would be a useful addition to the Arimaa gameroom. I still don't see how that makes me (or those other game servers and all the people on them, most of whom use formulas and ignore lists) into fascists, or how fascism is relevant to the issue.
Are I and Fairy welcome? Or should we leave?
[
The only one leaving is Arimanator.
Omar
]
|
| « Last Edit: May 25th, 2006, 8:24pm by Forum Admin » |
IP Logged |
|
|
|
|
Home
Help
Search
Members
Login
Register
Reply
Notify of replies
Send Topic
Print