Arimaa Forum - Print Page


    
      
        Arimaa Forum
        (http://arimaa.com/arimaa/forum/cgi/YaBB.cgi)
      

        Arimaa >> General Discussion >> Unrating Games / Integrity of Ratings
        
(Message started by: Fritzlein on May 25^th, 2006, 10:29am)

Title: Unrating Games / Integrity of Ratings
Post by Fritzlein on May 25^th, 2006, 10:29am

I remember the extreme frustration I felt when I was working through the bots of my day. When I lost games due to interface issues, exacerbated by my slow connection, it seemed so unfair I contemplated quitting Arimaa entirely. It is ridiculous for the ratings to reflect the speed and stability of one's connection to the server rather than one's ability to play Arimaa.

The solution, of course, is a more stable interface. Until a better interface could be developed, however, Omar offered the band-aid of unrating games lost due to interface and connection issues. In theory this could make the ratings more accurate, because players wouldn't lose rating points due to techinical issues on games they were otherwise winning.

Unfortunately, it has become common practice for players to unrate games they were otherwise losing. Even if the direct cause of a player losing the game is a disconnection or interface problem, to unrate a lost game that would probably be lost anyway makes the ratings less accurate, not more. Moreover, it appears probable that, in at least some cases, players have intentionally lost on time in positions that have become uncomfortable, merely to preserve the ability to make the game unrated. In short, the presence of Omar's band-aid is probably making the problem worse, rather than better.

I don't want to make a big fuss about ratings distortions due to retrospective unrating of games, because there are two problems that are much more serious. The first, as I've already mentioned, is the unstable interface. Any solution that doesn't improve the interface is bound to be problematic in some way.

The second problem is that ratings are going to be inaccurate anyway as long as the majority of games are human vs. bot games. It is common for players to inflate their ratings by incessantly playing bots they know how to beat. Also, since the introduction of the bot ladder, it has become common for players to deflate their ratings by incessantly playing bots they can't beat. Human vs. bot games demonstrate starkly how non-transitive the ratings are.

I've come to a point where I hardly care about human vs. bot games any more. Thirty-eight of my last fifty games have been against human opponents; the twelve bot games were mostly finishing up the bot ladder plus some stabs at bot-bashing that didn't hold my interest, so my percentage of games against bots may go down even further in the future. In this frame of mind, I'm much less concerned about the integrity of the ratings than I used to be, because the ratings are 90% a reflection of games versus bots. A year ago I inflated my rating to a record high by playing bots; now I am a better player (I hope!) but lower-rated. Who cares?

People who do care about ratings, however, should help Omar generate solutions to this problem. I can think of a few to get the discussion started:

1) Get rid of the feature for unrating games lost on time.

2) Keep the feature, but if anyone unrates a game they were losing, ban them from using the feature any more.

3) Instead of allowing one's most recent time loss to be unrated, allow it to be resumed from the final position. That would prevent people from avoiding uncomfortable positions. True, it would still allow people to gain extra thinking time in difficult positions, but this is a smaller issue that being able to dodge a lost position entirely. Also the player who won on time could be automatically compensated with (say) an extra three mintues of thinking time in reserve when the game resumes.

4) Put a timestamp in the interface, just as they have on the Internet Chess Club, so as to distinguish Internet lag from thinking for too long.

5) Just leave everything as is, accepting ratings distortions as a fact of life, but add a new category of rating, the "real" rating, which is based solely on human vs. human games. There is a greater chance of ultimately preserving the accuracy of a pure human vs. human rating, because there is more transitivity in human vs. human games, and because a percentage of the human vs. human games come from tournaments where the opponents are assigned, rather than self-selected.

I'm curious what everyone else thinks, but particularly curious what Omar is inclined to do about the issue, if anything.

Title: Re: Unrating Games / Integrity of Ratings
Post by RonWeasley on May 25^th, 2006, 12:28pm

Indeed, this is a topic people should comment on, but I didn't want to be the only one and seem like a ratings prefect. Thank you, Fritzlein, for taking the initiative.

I agree it is a Slytherin cheat to unrate a game lost due to play and not connection. Usually it is impossible to tell the difference, but now we have cases where players are admitting to it. It's weird, but I feel better about a person being honest, even about cheating. I had to stop and ask myself how I could feel this way.

The answer for me is that I would be more worried if the ratings had more integrity otherwise. However, anybody can get any rating they want through rated bot bashing. So if somebody manipulates their rating through timeouts, it's the same effect, just less honorable. If they admit to it, I know their rating isn't accurate, but for me it stops there. There is no increased risk of anybody cheating in muggle-muggle games because it can't be done.

One adverse effect is tournament seeding. This would be more serious if more money was on the line, but for me it's not enough to worry about. Again, if you want to manipulate your tournament seed you can do rated bot bashing (or sand bagging).

My inclination is to leave things as they are and allow people who want accurate ratings to behave honorably. People who break the honor system can enjoy a higher rating, but the quality of your play speaks for itself. This works both ways. For example, most of us know jdb plays much better than his rating. It's low, I think, because he prefers to play only hard opponents. This is an example of an inaccurate rating for a completely legitimate reason. Our ratings aren't fooling any of the experienced players.

Another point of view, however. is how arimaa looks to the outside world. If we get a reputation for cheating, say goodbye to any prospect of corporate sponsorship. Consider the value of your rating if everyone automatically assumes you cheated to get it. Nobody would take any of us seriously. Why would anybody new want to play? Hmm. I may have changed my mind.

Title: Re: Unrating Games / Integrity of Ratings
Post by jdb on May 25^th, 2006, 1:02pm

Personally, I don't worry about the ratings. Its fairly easy to tell how strong a person is by looking at a few of their recent games anyway.

Title: Re: Unrating Games / Integrity of Ratings
Post by Janzert on May 25^th, 2006, 1:26pm

I don't like the idea of people using this for reasons other than interface or server problems. Also some of the reasons given that I had seen recently did leave me feeling rather uncomfortable with them (and no don't even think I'm talking about you Arimanator none of these were your games I saw).

But at the same I'm not sure there is a practical fair way to handle it other than just living with it as it is now or shutting it off completely. I have a feeling at this stage shutting it off is probably going to be more detrimental overall than just living with it. At least until the interface/server is a little more stable for most players.

One thing that I think would help lessen the impact of skewed overall ratings would be seeing split out hvb, hvh and possibly seperatad time control bracket ratings.

On a side note, I do like 99of9's idea of only having the bot play quality games when rated and should I ever get a bot going will probably do the same.

Janzert

Title: Re: Unrating Games / Integrity of Ratings
Post by nbarriga on May 25^th, 2006, 2:19pm

I like the idea of different ratings, HvH, HvB and BvB, and maybe 2 or 3 different ratings based on time controls.

Also i strongly support the idea of only some games being rated, like tournament games, but that would require more frequent tournaments, maybe 4 times a year. All the tournaments need not be equal, you can have the postal one, the WC, a fast and a blitz one, and maybe some other ones.

Title: Re: Unrating Games / Integrity of Ratings
Post by chessandgo on May 25^th, 2006, 2:31pm

As jdb says, never mind the rating, at least as long as it doesn't come into account for the Worldchampionship and other tournaments' entrance, as it might be the case in the future when (if ?) the community grows to thousands of regular players.

As for Arimanator's complaints about PoTM, I think Omar made his point quite clear, and fully understandable : it is designed to promote h vs h games, and especially for newcomers who essentially play games vs bots through the ladder.
I have to say that at arimaa as well as at any other strategy game, playing stronger people and asking them advice is the better way to make progress. So the concept that a win on a beginner is as valuable as any other win is a good one to make the whole cummunity grow stronger by making a game with a lesser player attractive for the stronger as well.
In the last weeks a lot of stronger guys agreed to play with me and even challenged me sometimes, Nathan first. I owe them a lot.

So if your insults are directed at me, I just despise them, and if they're directed at Nathan, I lack words to tell you what I think of a guy who denies the great work accomplished by him.

As for 99's Gnobot playing well only in rated games, I don't understand the point, but he makes what he wants with it, had he not done it, we wouldn't be able to botbash it neither.

Jean

Title: Re: Unrating Games / Integrity of Ratings
Post by chessandgo on May 25^th, 2006, 4:00pm

I'm beginning to lack english words so I'd rather go on in french to be sure to stay within the limits of decency.

Sombre crétin ! Tu devrais envisager de garder tes conneries sur le PoTM, ce qui me permettrais de garder les miennes sur le botbashing. Pour ce qui est des insultes dont tu as été victime, aucune ne venait ni de moi ni de Nathan.

Ceci étant dit tu peux me dénigrer autant que tu veux, les insultes hors de l'échiquier étant tout à fait méprisables, surtout venant de quelqu'un qui n'a jamais joué avec moi. Que tu te permettes d'insulter Nathan par contre me met hors de moi.

Jean

Title: Re: Unrating Games / Integrity of Ratings
Post by PMertens on May 25^th, 2006, 4:45pm

#1 without a solution is unfair to those who think that rating is important

I dislike #2 because I prefer Omars way of NOT checking into games and just hoping that nobody is abusing this
In addition it can be really difficult to judge positions.

My favourite solution would be #3: the resume function.

I dislike #4 because "higher reasons" can be more than just disrupted connections ... why are phone calls (emergency or not) treated different ?
or a broken keyboard ?

I like #5 ... a hvh rating and a hvb rating are two very different informations (and I like additional infos)

Title: Re: Unrating Games / Integrity of Ratings
Post by unic on May 25^th, 2006, 5:25pm

on 05/25/06 at 10:29:49, Fritzlein wrote:

1) Get rid of the feature for unrating games lost on time.

I would definitely vote for this.

I also vote for not allowing players to force unrated games upon bots that don't want them.

If you want to fool around with silly handicaps, be ready to accept that your rating might take a hit.

If you care that much about your rating (... which I don't understand why anyone would ...) - then don't fool around. Or play against people or bots who are happy to play unrated games, and don't mind if you leave the game to time out or resign when it is barely off the ground.

It is very annoying for me, and a waste of time, to see Fairy play the start of an interesting game, and then the opponent leaves the game, and in fifteen minutes it times out, or the opponent resigns while there is still plenty of play left. Not having the possibility of forcing an unrated game would discourage such wasteful behaviour.

Fairy is not going to be available on the server for people to play against anymore, until I have found a way around this issue of people forcing unrated games on it... I wouldn't mind the games being unrated if people took them seriously - but now, it seems too easy to not take them seriously. Having all games rated would discourage such foolish behaviour.

Arimanator in particular did this several times, but others have done so as well. Therefore, Fairy is no longer available. It's a shame... I am grateful to those humans who played proper games until the end against it, whether they lost or won. I learned much about Fairy's weaknesses and strength from those games.

Finally, I would like to quote the section on sportsmanship, which is linked to from the gameroom:

Quote:

Try to avoid resigning

Although it is possible to resign a game of Arimaa, you should only use this option for practice or non-rated games. When you enter a serious game of Arimaa, think of it as a duel to the finish and never give up. Do not resign even if it looks like you are going to lose the game. Play it to the end. There are many examples of games where a player has actually won even though they were short on material. If you can pull off such a victory, it's even more exciting to watch.

And even if you can't pull of a victory, surely a good struggle is better than throwing the towel in at the slightest sign of a problem.

(Okay, I'll admit that I am far from perfect - I've resigned games against bots, especially when I've been somewhat in a hurry... but in theory, I do agree with what the above quote says.)

So, if you mean to play a serious game, don't resign. If you don't mean to play a serious game, don't play my bot and waste its and my time!

Title: Re: Unrating Games / Integrity of Ratings
Post by unic on May 25^th, 2006, 6:28pm

on 05/25/06 at 18:23:08, Arimanator wrote:

I won't frequent a place where some of the people wear a symbolic yellow star.

I wear a black Miscatonic University T-shirt, not a yellow star. ^^

Title: Re: Unrating Games / Integrity of Ratings
Post by unic on May 25^th, 2006, 6:49pm

(Who's not very bright and not recognizing sarcasm? ;) )

Well, chess servers and other game servers allow both formulas and ignore lists - it's a common and useful feature. See for example Internet Chess Club, Free Internet Chess Server, ChessBase's Playchess.com and the UoA's Generic Game Server, which all have some variation or other of this.

I don't see the problem with that. In fact, if Arimaa is to continue growing, something like formulas and being able to ignore certain people/bots would be very useful sooner or later. :)

Title: Re: Unrating Games / Integrity of Ratings
Post by unic on May 25^th, 2006, 7:07pm

(Apart from silly comments - I guess trying to keep the tone light didn't work...) As I pointed out, this is a common feature in many other game servers. I do not see what is wrong with it.

Title: Re: Unrating Games / Integrity of Ratings
Post by chessandgo on May 25^th, 2006, 7:41pm

Hum, I put back my clown hat that I shouldn't have put aside :P to remind you guys that facism and the Shoah and not funny matters and people might be offended seeing us speaking so lightly of it ... So I want hereby to apologize to all the fervent nazis for the lightness of what is being said here ! :)

Title: Re: Unrating Games / Integrity of Ratings
Post by unic on May 25^th, 2006, 7:46pm

[ this part of the post was deleted since it was a response to a post by Arimanator; and that post was also deleted ]

And as far as the annual challenge goes, I am not concerned with that, and was not planning to take part in it (for various reasons... mainly, I can't be bothered getting Fairy to run on Linux).

What I do plan to do is code a graphical interface to Fairy and release it, once I am happy with it. Then people can download it and play with it if they want to. Until then, I would prefer to have it play only games that are useful for me in learning more about Fairy and Arimaa - and if that means sticking to playing other bots, that is a sad state of things.

(And while we're at it - I had planned to wait with bringing this up until I actually started programming it - but what would people like to see in a graphical interface? I have downloaded the game databases, and was hoping to make them accessible in some face... compare the way the Othello program WZebra gives access to the Thor game databases. What other features would be viewed as useful?)

Title: Re: Unrating Games / Integrity of Ratings
Post by unic on May 25^th, 2006, 7:54pm

[ this part of the post was deleted ]

Omar, Fritzlein, Swynndla, Arimanator, everybody else? Should I remove *my* bot? Or should I keep it here? I will abide by what the majority say (I will count this by bedtime tomorrow night...). If people would rather not have Fairy here, there are plenty of other abstracts I would be curious to try programming an AI for... the only reason Arimaa was the one I actually chose was the access to easy testing via the server.

This is my final post on this topic. I still think accept/decline formulas and ignore lists (which are a standard function on many other game servers) would be a useful addition to the Arimaa gameroom. I still don't see how that makes me (or those other game servers and all the people on them, most of whom use formulas and ignore lists) into fascists, or how fascism is relevant to the issue.

Are I and Fairy welcome? Or should we leave?

[
The only one leaving is Arimanator.

Omar
]

Title: Re: Unrating Games / Integrity of Ratings
Post by Swynndla on May 25^th, 2006, 8:05pm

Unic, everyone else welcomes your bot. Please don't leave because of one person!

Title: Re: Unrating Games / Integrity of Ratings
Post by 99of9 on May 25^th, 2006, 8:08pm

Of course you are welcome unic, as is fairy.

I will discuss your suggestions of blacklists with you at a later date, when things are less heated.

Title: Re: Unrating Games / Integrity of Ratings
Post by chessandgo on May 25^th, 2006, 8:16pm

God, Unic !!!! Of course, don't leave ! and Fairy neither ... never forget that we're here for fun ...

By the way, I don't care if you always decline to play with players stronger than 2000, but if you move your top bar to 1990, I go away !!! :)

Jean

Title: Re: Unrating Games / Integrity of Ratings
Post by Swynndla on May 25^th, 2006, 8:31pm

heheh chessandgo, you'll be over 2000 very soon I'm sure ;)

Title: Re: Unrating Games / Integrity of Ratings
Post by omar on May 25^th, 2006, 9:16pm

Thanks for starting this topic Karl.

I recently had to send warning to a couple players about exessive use of the 'change game' option. I considered disabling that option, but it would be unfair to the majority of the players who use it properly. I could also disable the option for selected players, but I hate having to make those kind of decisions. But if someone is abusing it and enough different players send me a note about it I will disable the feature for that player.

As far as what to do about the inaccurate ratings. I think for now I will have to go with option #5. I still want to go back and revisit the rating system issues again when I get more time.

I would encourage people to download the games database and write their own programs to generate the *real* ratings. If the results of the program could be made available on the Web, I could link to it from the Arimaa site.

The different programs using different methods for computing the *real* ratings will come up with different rating values. We can then discuss these and suggest changes to the programs and possibly make progress towards a good rating system.

Title: Re: Unrating Games / Integrity of Ratings
Post by chessandgo on May 25^th, 2006, 9:24pm

on 05/25/06 at 19:54:24, unic wrote:

[ this part of the post was deleted ]

Are I and Fairy welcome? Or should we leave?

[
The only one leaving is Arimanator.

Omar
]

I see I'm not the only one to have a clown soul here :)

More seriously it's a pity to exclude a player indeed ... but I'm pretty sure you'll have even harder times in the future Omar ... :(

Title: Re: Unrating Games / Integrity of Ratings
Post by Fritzlein on May 25^th, 2006, 10:13pm

on 05/25/06 at 19:54:24, unic wrote:

[ this part of the post was deleted ]Should I remove *my* bot? Or should I keep it here?

I'm with everyone who would like to see more of Fairy. If you only want rated games, post a polite request to that effect to anyone who plays in unrated mode, and I think the vast majority of us will respect your wishes. The only question for me is how to persuade you to join the Computer Championship, because Fairy would make a great addition.

Title: Re: Unrating Games / Integrity of Ratings
Post by Fritzlein on May 25^th, 2006, 10:16pm

on 05/25/06 at 21:16:17, omar wrote:

I think for now I will have to go with option #5.

OK, this may inspire me to attempt a pure hvh rating system. As for the regular rating which stays the same, we'll all know what to think of it. ;-)

Title: Re: Unrating Games / Integrity of Ratings
Post by RonWeasley on May 26^th, 2006, 7:48am

I just got in this morning. Did I miss anything?

Title: Re: Unrating Games / Integrity of Ratings
Post by PMertens on May 26^th, 2006, 10:53am

not really ...

How about bots, that simply resign in unrated games ?
I also would like a greeting message like:

"Welcome back user. Since you resigned the last 3 games against me this is your last chance or I will not play you again :-P"

Title: Re: Unrating Games / Integrity of Ratings
Post by aaaa on May 26^th, 2006, 12:10pm

I know resigning against human players is frowned upon here, but what's wrong with doing it against bots? I do it all the time and I haven't heard as much as a peep from them.

Frankly, I don't understand what this whole obsession with ratings is all about. This is not go, where one's rating/ranking matters with respect to determining handicap.

Title: Re: Unrating Games / Integrity of Ratings
Post by chessandgo on May 26^th, 2006, 12:13pm

agreed !

Paul : I wish bots could chat !!!!

Title: Re: Unrating Games / Integrity of Ratings
Post by Fritzlein on May 26^th, 2006, 12:39pm

on 05/26/06 at 12:10:04, aaaa wrote:

I know resigning against human players is frowned upon here, but what's wrong with doing it against bots?

I don't think you are going to hurt a bot's feelings by resigning in general, but there's a specific case here that is different: Unic is actively developing Fairy, and he is putting it on line in order to get feedback on how well it is playing. I don't think unic cares about the rating either, he just wants "serious" games so the result is more useful. If people would play just as seriously against Fairy in unrated games as in rated games, I don't think there would be any issue for unic at all.

Quote:

Frankly, I don't understand what this whole obsession with ratings is all about. This is not go, where one's rating/ranking matters with respect to determining handicap.

Sure, so you also don't see the need for a "Change Game" feature? That is to say, it wouldn't bother you at all if the Omar removed the feature for unrating games lost because of technical problems? From the perspective of, "Let's not obsess about ratings", it makes much more sense to me to disable the feature than to leave it intact and ignore abuses.

Title: Re: Unrating Games / Integrity of Ratings
Post by Fritzlein on May 26^th, 2006, 12:43pm

Well, chrismccoll must not read the forum or the comments on his games. If he reads the comments, he must not care what anyone else thinks. He's at it again with game 32205.

http://arimaa.com/arimaa/gameroom/comments.cgi?gid=32205

As I said in my answer to aaaa, if we decide that the itnegrity of the ratings isn't such a big deal anyway (perhaps recognizing that ratings manipulation happens other ways as well), then I'd rather see the feature removed than see it abused.

Title: Re: Unrating Games / Integrity of Ratings
Post by aaaa on May 26^th, 2006, 12:59pm

on 05/26/06 at 12:39:56, Fritzlein wrote:

Sure, so you also don't see the need for a "Change Game" feature? That is to say, it wouldn't bother you at all if the Omar removed the feature for unrating games lost because of technical problems?

Does the fact I even resigned games only because the bot timed out answer your question?

Anyway, Flash tends to crash my browser once in a while, so I would appreciate the existence of another client. It could even be another online one, except that it's programmed in, say, Java.

Title: Re: Unrating Games / Integrity of Ratings
Post by unic on May 26^th, 2006, 1:13pm

on 05/26/06 at 12:39:56, Fritzlein wrote:

Well summarized by Fritzlein. I want games to be serious and played until the end, so that they are useful to me to in trying to improve Fairy. I am often sitting watching the games live and studying Fairy's output as it is playing.

Title: Re: Unrating Games / Integrity of Ratings
Post by PMertens on May 26^th, 2006, 2:26pm

Quote:

I wish bots could chat !!!!

Actually as far as I know there is the possibility of implementing chat to a bot.
Especially making a startup-message should be quite easy ... if I understood that perl-script correctly ;-)

Quote:

Frankly, I don't understand what this whole obsession with ratings is all about. This is not go, where one's rating/ranking matters with respect to determining handicap.

I honestly do not care about ratings ... this is just a game to me ...

for the resign thing:
I guess Fritzl said nearly everithing there is to say ...

Anyway I doubt that resigning is such a problem ... the unrate button is worse because people start playing strange experimental things, but stop when it gets interesting ...

Please Unic put it back online .. I will happily train it (sometimes when I am online ... since I am still on leave ;-) )

Title: Re: Unrating Games / Integrity of Ratings
Post by chessandgo on May 26^th, 2006, 6:06pm

on 05/26/06 at 12:43:11, Fritzlein wrote:

The fact that Fritz picked up the game among thousands of correct ones shows his omniscience.

This is not only another proof of Godel's theorem stating the existence of god, but also a constructive one !! Still Goedel didn't manage to prove god's uniqueness, so there is still hope to prove that Robinson is one as well :)

More seriously (or not ...) never mind that someone makes bad use of the feature ; not only it didn't make his rating go up but merely prevented it to go down (and down by 1 point or 2 I guess ;) ), but there's is still 900 points between his rating and yours Fritz :) And we all know you made it without deception !!!

jean

Title: Re: Unrating Games / Integrity of Ratings
Post by PMertens on May 27^th, 2006, 5:38am

it annoys me much more that everytime I want to take a look at the last commented games I have to scroll away some junk ;-)

Title: Re: Unrating Games / Integrity of Ratings
Post by chessandgo on May 27^th, 2006, 11:38am

too true !! Basically when you look at the "recent commented games", and given the server instability these days, you see at least 45 "This game was changed to unrated blabla" out of 50 ; so maybe someone has added a very interesting comment on a very interesting game sooner the same day, but you are going to miss it anyway ...

The possibility of adding comment to a game is one of the very VERY nice features of this website ; maybe we should ask Omar to remove the request to add a comment at changed to unrated games ?
For instance we wouldn't miss anymore the last comment about the last Fritzlein-PMertens game ? ;)

Title: Re: Unrating Games / Integrity of Ratings
Post by 99of9 on May 27^th, 2006, 6:13pm

I disagree with removing the "unrate game" comment. That is the only way the community is able to watch for abuse, and see players reasons for abandoning.

I agree there are too many of these comments at the moment, but that is just because the feature is used too often at the moment (not all because of server instability).

Title: Re: Unrating Games / Integrity of Ratings
Post by 99of9 on Jun 1^st, 2006, 7:59am

on 05/26/06 at 12:43:11, Fritzlein wrote:

Well, chrismccoll must not read the forum or the comments on his games. If he reads the comments, he must not care what anyone else thinks.

This is continuing. I don't know how we can contact him if he doesn't read his comments. Omar perhaps you could directly send an email asking him to stop?