Author |
Topic: World Championship tournament format (Read 9308 times) |
|
Fritzlein
Forum Guru
Arimaa player #706
Gender: 
Posts: 5928
|
 |
Re: World Championship tournament format
« Reply #15 on: May 18th, 2005, 8:48pm » |
 Quote  Modify
|
I've thought somewhat about the conflict between "every player should have a chance" and "the truly strongest player should win".
First, I think that for the World Championship to be as legitimate as possible, it should be open to everyone. Specifically, qualifying for the tournament should not be based on ratings, because of the flaws inherent in the rating system. At the moment ratings measure mostly how well one does against bots rather than against humans. One dramatic example of how ratings could exclude a legitimate contender is that Naveed is currently ranked 16th among humans! If the World Championship were open only to the top 8 (by rating) who signed up, Naveed might not get a seat despite having scored victories against every other top player at one time or another.
On the other hand, although I believe everyone should have a chance to participate, I don't like formats which are conducive to upsets. Single-elimination is too vulnerable to freak result. Once everyone has signed up, the tournament should have as great a chance as possible of declaring the best player to be the winner.
Obviously, the longer the tournament is allowed to run, the greater the chance that the best player wins. However, even if you fix a certain number of rounds and declare that the tournament can't go any longer than that maximum, within that time frame some formats will have a greater chance than others that the best player will win. My contention is that floating triple-elimination is nearly as short as traditional double-elimination, but more fair and more likely to leave the best player standing.
One final thought about time commitment: If the main problem with a large number of rounds is that some people might not be able to commit to a game per week for X straight weeks, folks might be able instead to commit to one game every two weeks over a total of 2X weeks. That gives a longer time between tournament begin and end, which is undesirable, but takes a lot of pressure off the players, which might make it worth it.
|
| « Last Edit: May 19th, 2005, 1:21pm by Fritzlein » |
 IP Logged |
|
|
|
omar
Forum Guru
Arimaa player #2
Gender:
Posts: 1003
|
|
Re: World Championship tournament format
« Reply #16 on: May 19th, 2005, 7:22pm » |
Quote Modify
|
I forgot to mention one of the benifits of having seperate tournaments. In addition to using different formats the different tournaments could also use different time controls. We've always had a conflict with time controls as well. Faster time controls allow more people to be able to participate, but for WC quality games we want to use longer time controls. The Open Classic could use a time controls like 1 min per move while the WC could use time controls of say 2 min per move. I think finding a long streatch of free time during a week to fit in a game has been the main problem for most people. So using faster time controls for the OC could help eliminate that.
I tried to search the Internet for a quantatative comparison of different tournament formats. Particularly with respect to a given formats ability to select the player with the true highest rating. But I didn't find anything even close to it. If someone knows of such previous work please let me know.
I did however come across this interesting article:
http://www.johnpratt.com/items/docs/ranker/ranker.html
It suggests that all players in a tournament should be ranked to make it more interesting for everyone. Maybe a format like this could be used for the OC.
I think having a WC tournament that is limited to just a select number of top rated players can be considered the final stage of a much longer tournament that was open to everyone. If it is known well in advance that only the 4 top rated players will continue into the finals of the WC then a player who is really interested to win the WC has basically the whole year to get their ratings up. Of course we could very easily setup a page to compute the ratings based only on H-H games and select the top players from that page to avoid people inflating their ratings against bots. Although there are some flaws with rating systems, the ratings are still a very good predictor on a gross level. It is only when the rating difference between players is small that ratings are not accurate. So I think it is good to make use of ratings as a way of narrowing the field for the final stage of the WC.
|
|
IP Logged |
|
|
|
jdb
Forum Guru
Arimaa player #214
Gender:
Posts: 682
|
|
Re: World Championship tournament format
« Reply #17 on: May 20th, 2005, 8:30am » |
Quote Modify
|
Here is an article about the effectiveness of different types of tournaments. It gave me a headache, but it might be what your looking for.
http://www.chessbase.com/newsdetail.asp?newsid=260
|
|
IP Logged |
|
|
|
omar
Forum Guru
Arimaa player #2
Gender:
Posts: 1003
|
|
Re: World Championship tournament format
« Reply #18 on: May 21st, 2005, 12:54am » |
Quote Modify
|
Thanks for that link Jeff; it was very interesting. Looks like Jeff Sonas actually ran thousands of simulations on 13000 tournament formats using simulated mearsured and true ratings for the players. This is exactly the kind of data we need to look at make a more informed decision. I just wish that the full results were published along with a more precise description of how the simulations were run.
Never the less, it is interesting that none of the formats even break the 70% mark for selecting the true strongest player. And these are formats that have long series of about 20 games between the final candidates.
After reading Jeff's article I get the feeling that there probably isn't that much of a difference between the single, double and triple elimination formats with respect to selecting the best player as the winner. But we really should run some simulations and see what the numbers are. Suppose we generate 16 players with true ratings randomly assigned between the range of 1500 to 2000. Then each players measured rating is produced by adding a random number between -50 to 50 to the true rating. We simulate a tournament by using the true ratings to determine the outcome of the games and use the mearsued ratings as needed for pairing. After the tournament is over we check to see if the player with the highest true rating has won the tournament. After running thousands of such simulations we can get a pretty good idea of how good a particular tournament format is at selecting the best player.
It would be great if someone could try this out for the single, double and triple elimination formats. Any volunteers ?
|
|
IP Logged |
|
|
|
Fritzlein
Forum Guru
Arimaa player #706
Gender:
Posts: 5928
|
|
Re: World Championship tournament format
« Reply #19 on: May 21st, 2005, 1:13am » |
Quote Modify
|
on May 21st, 2005, 12:54am, omar wrote:| Suppose we generate 16 players with true ratings randomly assigned between the range of 1500 to 2000. Then each players measured rating is produced by adding a random number between -50 to 50 to the true rating. |
|
It might be more realistic to add a random number between -250 and +250. There are many examples of wildly inaccurate ratings, including Arimanator jumping from just over 1600 to just over 1800 within two days. Do you suppose his true playing strength changed 200 points overnight? I don't, and I conclude that at least one of the two ratings is inaccurate, if not both. Plus or minus fifty doesn't cover it, and it doesn't cover the inaccuracies in the ratings of lots of other players either, myself included.
|
|
IP Logged |
|
|
|
99of9
Forum Guru
Gnobby's creator (player #314)
 
Gender:
Posts: 1413
|
|
Re: World Championship tournament format
« Reply #20 on: May 21st, 2005, 2:12am » |
Quote Modify
|
on May 21st, 2005, 12:54am, omar wrote:
Never the less, it is interesting that none of the formats even break the 70% mark for selecting the true strongest player. And these are formats that have long series of about 20 games between the final candidates.
|
|
This depends on how close together the top few players are in real ratings. If one player was far ahead, nearly all tournament formats would select the correct player. If all players were about equal, in a 16 player tournament, most tournament formats would only give the correct winner just over 1/16 of the time.
The distribution of top chess players is probably different to the distribution of top arimaa players, since we do not have any professionals.
Guessing a distribution such as that omar suggests (16 randomly spaced between 1500 and 2000) is probably going to introduce significant biases in these calculations. It would be better to use actual numbers from current ratings or previous tournaments.
But to be honest I think programming in the details of each tournament method sounds like more work than the results will be worth. I think most of the guestimated comments in this thread about the effects of different tournament methods (and their lengths) are fairly accurate.
My vote is still for the double elimination variant that Robert and I were talking about. It's not as long as Fritz's triple-elim, and it has significantly higher accuracy than knockout or traditional double-elim.
Regarding having 2 different tournaments, with candidature decided by ratings. I think that would be open to serious exploitation if the bots were involved in the ratings. Even if it was cut down to human-only ratings, collusion may become an issue (eg if one group of friends decided to all help one candidate get in). The other problem with using ratings is that it encourages very high rated players to stop playing often, in case their rating takes a dive. On Fritz's current rating, he could be in the tournament for the next 20 years without playing a game!
|
|
IP Logged |
|
|
|
99of9
Forum Guru
Gnobby's creator (player #314)
Gender:
Posts: 1413
|
|
Re: World Championship tournament format
« Reply #21 on: May 21st, 2005, 5:46am » |
Quote Modify
|
Naveed is quite an unusual case though, because his real on-the-day playing quality varies so much. If he's been away from the game for quite some time, or is just having a bad day, he often has a *very* bad day - and can overlook threats from even the simplest bots. When in form though, he is one of the fiercest competitors and may beat any of us. I think this may be why he had trouble in the postals... some days his moves were very good, others were very bad. So I don't think even limiting it to H-H will fix Naveed's rating oscillations!
It would be nice to still have the bots involved in the ratings lists though.. just so we know how humanity is doing against cyberdom. I think humans often exaggerate how far humanity is in front of cyberdom at the moment - and the ratings lists are often a small antidote to our delusions.
One possibility is that any game *against* a human should count. That way a Human v Bot game, would count for the bot but not for the human. Unfortunately the bots might get an unfair rating inflation if humans started taking these games less seriously.
I support the idea of having this kind of rating system, but personally I'd say it'd be better to run both at once - if it's not too complicated for Omar.
|
| « Last Edit: May 21st, 2005, 5:49am by 99of9 » |
IP Logged |
|
|
|
omar
Forum Guru
Arimaa player #2
Gender:
Posts: 1003
|
|
Re: World Championship tournament format
« Reply #22 on: May 21st, 2005, 10:04am » |
Quote Modify
|
on May 21st, 2005, 5:46am, 99of9 wrote:
I support the idea of having this kind of rating system, but personally I'd say it'd be better to run both at once - if it's not too complicated for Omar. |
|
Our previous discussions about rating sytems are still pending.
rating inflation/deflation:
http://arimaa.com/arimaa/forum/cgi/YaBB.cgi?board=talk;action=display;nu m=1065901453;start=0
distortion due to selection of opponents:
http://arimaa.com/arimaa/forum/cgi/YaBB.cgi?board=talk;action=display;nu m=1096120807;start=
distortion due to time controls:
http://arimaa.com/arimaa/forum/cgi/YaBB.cgi?board=talk;action=display;nu m=1103741634;start=
I hope to get back to those eventually and hopefully completely revamp the rating system based on what we come up with.
We have made good progress on it so far by identifying the problems and discussed some possible solutions. The three main problems areas are:
* rating inflation/deflation due to a floating scale
* rating inaccuracies due to selection of opponents
* rating inaccuracies due to time controls
We've discssed possible solutions to each, but it's still in progress.
For the puropose of this discussion, lets just assume the problems with the rating system have been fixed.
|
| « Last Edit: Jun 15th, 2005, 6:53am by omar » |
IP Logged |
|
|
|
omar
Forum Guru
Arimaa player #2
Gender:
Posts: 1003
|
|
Re: World Championship tournament format
« Reply #23 on: May 21st, 2005, 10:09am » |
Quote Modify
|
on May 21st, 2005, 1:13am, Fritzlein wrote:
It might be more realistic to add a random number between -250 and +250. There are many examples of wildly inaccurate ratings, including Arimanator jumping from just over 1600 to just over 1800 within two days. Do you suppose his true playing strength changed 200 points overnight? I don't, and I conclude that at least one of the two ratings is inaccurate, if not both. Plus or minus fifty doesn't cover it, and it doesn't cover the inaccuracies in the ratings of lots of other players either, myself included. |
|
Actually once we have simulation program setup, it will be very easy to change high level parameters like rating inaccuracies and see what the new results are. So we can definitely try out +-250 and maybe more.
|
|
IP Logged |
|
|
|
omar
Forum Guru
Arimaa player #2
Gender:
Posts: 1003
|
|
Re: World Championship tournament format
« Reply #24 on: May 21st, 2005, 11:03am » |
Quote Modify
|
on May 21st, 2005, 2:12am, 99of9 wrote:
Guessing a distribution such as that omar suggests (16 randomly spaced between 1500 and 2000) is probably going to introduce significant biases in these calculations. It would be better to use actual numbers from current ratings or previous tournaments.
|
|
We can vary the distribution range as well and see how that effects the results. But keep in mind that we are not interested in the result of any one format, but rather a comparision between formats. So if format A is better than format B for the range I proposed, it will probably be better on a different range as well.
Quote:
But to be honest I think programming in the details of each tournament method sounds like more work than the results will be worth. I think most of the guestimated comments in this thread about the effects of different tournament methods (and their lengths) are fairly accurate.
|
|
Well the problem with basing the decision on guestimats rather than actual figures is that you never know for sure if your decision is right or not. Suppose a year later someone proposes a new format for the WC; we will be stuck trying to defend what we selected based only on guestimates and have no way to compare it against any new proposals. I think the effort we put in now to quantatively compare some tourn. formats and make an informed decision will be worthwhile in the long run. If a new proposal comes along in the future we can objectively compare it and possibly accept it without being biased towards our current system.
Quote:
My vote is still for the double elimination variant that Robert and I were talking about. It's not as long as Fritz's triple-elim, and it has significantly higher accuracy than knockout or traditional double-elim.
|
|
Hummm, how much is "significantly higher" 
Since there doesn't seem to be any volunteers to do this, I'll start on it.
|
|
IP Logged |
|
|
|
Fritzlein
Forum Guru
Arimaa player #706
Gender:
Posts: 5928
|
|
Re: World Championship tournament format
« Reply #25 on: May 21st, 2005, 11:49pm » |
Quote Modify
|
One guesstimate that I highly expecte simulation to prove true is that pairings of the form
1 v 16
2 v 15
3 v 14
...
8 v 9
give a greater chance of the truly strongest player winning overall than do pairings of the form
1 v 9
2 v 10
3 v 11
...
8 v 16
as we had in the last World Championship. But I am guessing that you had a well-considered reason for doing it the latter way, Omar. Do you mind explaining the rationale for doing the pairings as you did? Perhaps there are factors we are forgetting in the quest to have a tournament structure that is more likely to have the strongest player win overall.
|
|
IP Logged |
|
|
|
omar
Forum Guru
Arimaa player #2
Gender:
Posts: 1003
|
|
Re: World Championship tournament format
« Reply #26 on: May 22nd, 2005, 10:25am » |
Quote Modify
|
on May 21st, 2005, 11:49pm, Fritzlein wrote:| But I am guessing that you had a well-considered reason for doing it the latter way, Omar. Do you mind explaining the rationale for doing the pairings as you did? Perhaps there are factors we are forgetting in the quest to have a tournament structure that is more likely to have the strongest player win overall. |
|
Don Dailey had suggested sliding the lower half to play the upper half. He thought it would work well because it would eliminate the most number of weak players at each round. He had also mentioned the folding method would work better, but said it relied heavily on the ratings being accurate. So with no actual figures to go by and knowing that our rating system is new and based on a small pool, I decided to go with the sliding method. Once we have a system to compare tournament formats we'll be able to test your hypothises and know for sure if it is better and exactly how much.
|
|
IP Logged |
|
|
|
omar
Forum Guru
Arimaa player #2
Gender:
Posts: 1003
|
|
Re: World Championship tournament format
« Reply #27 on: May 22nd, 2005, 10:32am » |
Quote Modify
|
on May 21st, 2005, 1:13am, Fritzlein wrote:
Plus or minus fifty doesn't cover it, and it doesn't cover the inaccuracies in the ratings of lots of other players either, myself included. |
|
Karl, while Im working on the programs for comparing tournament systems, could you please do some kind of analysys with the games archive to see how well the ratings are predicting outcome of games and see if that can be converted to a number we can use for rating inaccuracies in the simulations.
|
|
IP Logged |
|
|
|
omar
Forum Guru
Arimaa player #2
Gender:
Posts: 1003
|
|
Re: World Championship tournament format
« Reply #28 on: May 22nd, 2005, 10:48am » |
Quote Modify
|
on May 21st, 2005, 2:12am, 99of9 wrote:
Guessing a distribution such as that omar suggests (16 randomly spaced between 1500 and 2000) is probably going to introduce significant biases in these calculations. It would be better to use actual numbers from current ratings or previous tournaments.
|
|
If we use measured ratings from the previous tournaments, what should we do about the true ratings. Should we just add a random number in the range of the rating inaccuracies to the measured rating and use that. Should the true ratings be changed from one trial to the next, or computed just once and used for all trials?
|
|
IP Logged |
|
|
|
99of9
Forum Guru
Gnobby's creator (player #314)
Gender:
Posts: 1413
|
|
Re: World Championship tournament format
« Reply #29 on: May 22nd, 2005, 6:46pm » |
Quote Modify
|
on May 22nd, 2005, 10:25am, omar wrote:
Don Dailey had suggested sliding the lower half to play the upper half. He thought it would work well because it would eliminate the most number of weak players at each round. He had also mentioned the folding method would work better, but said it relied heavily on the ratings being accurate. So with no actual figures to go by and knowing that our rating system is new and based on a small pool, I decided to go with the sliding method. Once we have a system to compare tournament formats we'll be able to test your hypothises and know for sure if it is better and exactly how much.
|
|
One thing is clear about the sliding system, with no simulations necessary. In every round the top rated player has to play a harder opponent than the second rated player, until they meet. This creates a clear bias toward the chances of the 2nd top player winning the tournament.
In fact even Don's reasoning doesn't make sense to me. Take the round of 16 for example. In the sliding method, #8 plays #16, but #9 plays #1. This is a massive reinforcement of small anomalies that may occur in the ratings of #8 and #9. I agree that it "will knock out more of the lower rated players", but not neccessarily by real rating, just by predicted rating. It seems to me much fairer for #8 to play *against* #9 in that round to get a rough determination of who in fact is the better player that will get to contest the next round of the tournament. That is what would happen in the crossover method.
|
|
IP Logged |
|
|
|
|
Home
Help
Search
Members
Login
Register
Reply
Notify of replies
Send Topic
Print