Author |
Topic: rating player strength by community voting (Read 1284 times) |
|
froody
Forum Guru
I <3 nurpinar
  
Gender: 
Posts: 103
|
 |
rating player strength by community voting
« on: Dec 14th, 2011, 4:53pm » |
 Quote  Modify
|
What we're striving for is a way to rank players in terms of playing strength. This is not easy and any system based on applying mathematical rules to results is going to be open to abuse/weaknesses. It's a difficult problem, so why not take advantage of human intelligence and simply ask the community to decide what is a fair rating for each player. We all have a pretty good idea about how strong each player is just by watching their games. We all know hanzack is a bad arimaa baby and resigns instead of goaling because of his fear of numbers. We all know which players have artificially boosted their rating by bot bashing. We all know which players are under-rated because they're working on beating a strong bot and losing a lot. We all know which players are afraid to play rising stars because they care about their rating and are afraid to lose. Why not take advantage of this intelligent knowledge and get people to vote on how strong we think each player is?
How exactly should this voting system work? I don't know. Suppose we each rank the top x players and say what ratio of games we guess they will win against against other players? Then do some math. Does that work? We could also end up with a much more meaningful number that tells people exactly how many times better one player is than another. All the votes will be public.
Is this open to abuse/weaknesses? Has it been discussed before? Has it been tried with other games?
|
|
 IP Logged |
“Arimaa holds its master in its own bonds, shackling the mind and brain so that the inner freedom of the very strongest must suffer”
|
|
|
novacat
Forum Guru
Arimaa player #751
Gender:
Posts: 119
|
|
Re: rating player strength by community voting
« Reply #1 on: Dec 15th, 2011, 7:20am » |
Quote Modify
|
The first thing that comes to mind is the BCS standings that rank US college (American) football. Two human polls and 4 computerized algorithms are used to determine a team's ranking. There is much debate about this system.
As for Arimaa, the WHR rating does a good job of rating players and takes away the bot related issues. I don't know that humans are any better than formulas at getting the rating right in most cases. We can't really get a good feel for a player's strength unless they play a lot of games. The ratings formula also does a good job when it has a lot of samples. There are a few exceptions, such as hanzack.
Of course, even with humans choosing the rating, there will be lots of disagreement, and we will need a formula to take everyone's input and come up with one result. 
|
|
IP Logged |
|
|
|
froody
Forum Guru
I <3 nurpinar
Gender:
Posts: 103
|
|
Re: rating player strength by community voting
« Reply #2 on: Dec 15th, 2011, 11:08pm » |
Quote Modify
|
Does not the wisdom of crowds apply here?
I think it would be nice to have several different systems and make them all visible in the gameroom. You can never have too much information.
How would you do the voting thing? How would we measure which system was predicting results more accurately?
|
|
IP Logged |
“Arimaa holds its master in its own bonds, shackling the mind and brain so that the inner freedom of the very strongest must suffer”
|
|
|
leo
Forum Guru
Gender:
Posts: 278
|
|
Re: rating player strength by community voting
« Reply #3 on: Dec 17th, 2011, 12:33pm » |
Quote Modify
|
Is not the wisdom of crowds two-edged? 
However having several systems side to side sounds good to me.
|
|
IP Logged |
|
|
|
omar
Forum Guru
Arimaa player #2
Gender:
Posts: 1003
|
|
Re: rating player strength by community voting
« Reply #4 on: Dec 17th, 2011, 8:51pm » |
Quote Modify
|
on Dec 14th, 2011, 4:53pm, froody wrote:What we're striving for is a way to rank players in terms of playing strength. This is not easy and any system based on applying mathematical rules to results is going to be open to abuse/weaknesses. It's a difficult problem, so why not take advantage of human intelligence and simply ask the community to decide what is a fair rating for each player.
|
|
The mathematics of rating systems are pretty sound and if the games are of good quality the rating systems work very well. The problem is obtaining a good set of games. When people sandbag or play the same bots over and over to inflate their ratings, they are not directly messing up the rating system, they are indirectly messing it up by giving it games that are not played according to the assumptions of the rating system.
My guess would be that a rating system based on community voting would not produce very good ratings. It would also not scale very well if there were a large number of players.
However, it might work better if rather than the community voting on a players strength, the community voted on what games would be included in computing the ratings. Then a standard rating system could be applied to a better set of games and produce more accurate ratings. Having a program automatically decide if a particular game (which is already marked as rated) should really be considered rated or not is an AI problem. I attempted to write an algorithm once to sort out the good games from the bad ones and found that it was not easy. One has to look at not only the players histories and past behavior, but also the actual positions and moves of the games.
This is something that human intelligence could be applied to. So initially every rated game is considered rated, but if members of the community start voting that it should be unrated then it eventually becomes 'disliked' and is not used for computing the ratings.
I could add an 'unrate this game' link on the game comments page and show how many people have 'disliked' this game being a rated game. The disliked number could be exported to the games archive and rating and stats software could make use of it. If it gets abused then we could limit it to only senior members of the community being able to mark down the games.
|
| « Last Edit: Dec 17th, 2011, 8:58pm by omar » |
IP Logged |
|
|
|
|
Home
Help
Search
Members
Login
Register
Reply
Notify of replies
Send Topic
Print