Arimaa Forum - Print Page


    
      
        Arimaa Forum
        (http://arimaa.com/arimaa/forum/cgi/YaBB.cgi)
      

        Arimaa >> Off Topic Discussion >> Chess rating competition
        
(Message started by: Janzert on Aug 3^rd, 2010, 11:23am)

Title: Chess rating competition
Post by Janzert on Aug 3^rd, 2010, 11:23am

A competition similar in style to the netflix competition is being held to find rating systems that perform better than elo.

http://kaggle.com/chess

Janzert

Title: Re: Chess rating competition
Post by Fritzlein on Aug 3^rd, 2010, 2:20pm

"Competitors train their rating systems using a training dataset of over 65,000 recent results for 8,631 top players. Participants then use their method to predict the outcome of a further 7,809 games."

Seems to me the dataset is on the small side. For Netflix the training data was 100,480,507 ratings that 480,189 users gave to 17,770 movies, for the prediction of 2,817,131 further data points.

Title: Re: Chess rating competition
Post by omar on Aug 21^st, 2010, 2:02am

Thanks for sharing this with us Brian. I wonder if Remi knows about this competition. It would be interesting to see how WHR does in this. I'll check with him. Also I would be interested to try out the winning entry on our site. There is mention that the methodology must be shared in order to receive the prize.

Maybe I should post the Arimaa challenge here as a long term contest. I'll see what the terms are.

Title: Re: Chess rating competition
Post by omar on Aug 21^st, 2010, 2:07am

It doesn't look like the Arimaa Challenge would fit the format and time frame of their contest requirements.

Title: Re: Chess rating competition
Post by omar on Aug 22^nd, 2010, 12:29pm

Remi said he won't have time to work on entering WHR in this contest, but he is OK with it if we do.

Title: Re: Chess rating competition
Post by omar on Oct 7^th, 2010, 9:05am

I entered the Arimaa gameroom rating system into this contest http://kaggle.com/chess

It seems the additional rating uncertainty parameter isn't helping since the Elo Benchmark is still doing better. Elo has a RMSE of 0.707638 and the Arimaa gameroom rating system has a RMSE of 0.715348. The RMSE calculation is a bit out of the ordinary though since the games of each player are being grouped together by month before calculating the RMSE http://kaggle.com/chess?viewtype=evaluation

I don't know if any of the other competitors have already implemented WHR (can't tell from the names of the entries), but I would encourage woh or aaaa to run your implementation of WHR on the test data and submit the results to see how it does.

Title: Re: Chess rating competition
Post by omar on Oct 15^th, 2010, 11:06am

Jeff Sonas posted a PDF http://bit.ly/9aNOpJ

about the chessmetrics rating system that he uses on his site
http://www.chessmetrics.com/

I was able to implemented a simplified version of it and was impressed how well it performed in the rating system competition. It scored an RMSE of 0.674736 and put me at number 33 out of 213 on the Leaderboard. I'm not trying to win this contest (probably couldn't even if I tried), but following it closely to see how rating systems compare. My simplified version of the chessmetrics rating system did almost as well as WHR implemented by Ron Britvich (team Intuition) which is at position 32 with a score of 0.673922. As a comparison the Arimaa gameroom rating system scored 0.715348. Predicting all draws gets a score of 0.792134 (yes, I tried it :-) ).

Here is my simplified implementation of the chessmetrics rating system:
http://arimaa.com/arimaa/rating/chessmetrics/cmrs.txt

Feel free to modify it and make your own system.