Author |
Topic: Chess rating competition (Read 1072 times) |
|
Janzert
Forum Guru
Arimaa player #247
Gender: 
Posts: 1016
|
 |
Chess rating competition
« on: Aug 3rd, 2010, 11:23am » |
 Quote  Modify
|
A competition similar in style to the netflix competition is being held to find rating systems that perform better than elo.
http://kaggle.com/chess
Janzert
|
|
 IP Logged |
|
|
|
Fritzlein
Forum Guru
Arimaa player #706
Gender:
Posts: 5928
|
|
Re: Chess rating competition
« Reply #1 on: Aug 3rd, 2010, 2:20pm » |
Quote Modify
|
"Competitors train their rating systems using a training dataset of over 65,000 recent results for 8,631 top players. Participants then use their method to predict the outcome of a further 7,809 games."
Seems to me the dataset is on the small side. For Netflix the training data was 100,480,507 ratings that 480,189 users gave to 17,770 movies, for the prediction of 2,817,131 further data points.
|
|
IP Logged |
|
|
|
omar
Forum Guru
Arimaa player #2
Gender:
Posts: 1003
|
|
Re: Chess rating competition
« Reply #2 on: Aug 21st, 2010, 2:02am » |
Quote Modify
|
Thanks for sharing this with us Brian. I wonder if Remi knows about this competition. It would be interesting to see how WHR does in this. I'll check with him. Also I would be interested to try out the winning entry on our site. There is mention that the methodology must be shared in order to receive the prize.
Maybe I should post the Arimaa challenge here as a long term contest. I'll see what the terms are.
|
|
IP Logged |
|
|
|
omar
Forum Guru
Arimaa player #2
Gender:
Posts: 1003
|
|
Re: Chess rating competition
« Reply #3 on: Aug 21st, 2010, 2:07am » |
Quote Modify
|
It doesn't look like the Arimaa Challenge would fit the format and time frame of their contest requirements.
|
|
IP Logged |
|
|
|
omar
Forum Guru
Arimaa player #2
Gender:
Posts: 1003
|
|
Re: Chess rating competition
« Reply #4 on: Aug 22nd, 2010, 12:29pm » |
Quote Modify
|
Remi said he won't have time to work on entering WHR in this contest, but he is OK with it if we do.
|
|
IP Logged |
|
|
|
omar
Forum Guru
Arimaa player #2
Gender:
Posts: 1003
|
|
Re: Chess rating competition
« Reply #5 on: Oct 7th, 2010, 9:05am » |
Quote Modify
|
I entered the Arimaa gameroom rating system into this contest http://kaggle.com/chess
It seems the additional rating uncertainty parameter isn't helping since the Elo Benchmark is still doing better. Elo has a RMSE of 0.707638 and the Arimaa gameroom rating system has a RMSE of 0.715348. The RMSE calculation is a bit out of the ordinary though since the games of each player are being grouped together by month before calculating the RMSE http://kaggle.com/chess?viewtype=evaluation
I don't know if any of the other competitors have already implemented WHR (can't tell from the names of the entries), but I would encourage woh or aaaa to run your implementation of WHR on the test data and submit the results to see how it does.
|
|
IP Logged |
|
|
|
omar
Forum Guru
Arimaa player #2
Gender:
Posts: 1003
|
|
Re: Chess rating competition
« Reply #6 on: Oct 15th, 2010, 11:06am » |
Quote Modify
|
Jeff Sonas posted a PDF http://bit.ly/9aNOpJ
about the chessmetrics rating system that he uses on his site
http://www.chessmetrics.com/
I was able to implemented a simplified version of it and was impressed how well it performed in the rating system competition. It scored an RMSE of 0.674736 and put me at number 33 out of 213 on the Leaderboard. I'm not trying to win this contest (probably couldn't even if I tried), but following it closely to see how rating systems compare. My simplified version of the chessmetrics rating system did almost as well as WHR implemented by Ron Britvich (team Intuition) which is at position 32 with a score of 0.673922. As a comparison the Arimaa gameroom rating system scored 0.715348. Predicting all draws gets a score of 0.792134 (yes, I tried it  ).
Here is my simplified implementation of the chessmetrics rating system:
http://arimaa.com/arimaa/rating/chessmetrics/cmrs.txt
Feel free to modify it and make your own system.
|
|
IP Logged |
|
|
|
|
Home
Help
Search
Members
Login
Register
Reply
Notify of replies
Send Topic
Print