Author |
Topic: 2006 WC Prediction Contest (Read 5354 times) |
|
Ryan_Cable
Forum Guru
Arimaa player #951
Gender: 
Posts: 138
|
 |
Re: 2006 WC Prediction Contest
« Reply #1 on: Nov 14th, 2005, 8:35pm » |
 Quote  Modify
|
Thanks 
|
|
 IP Logged |
|
|
|
Fritzlein
Forum Guru
Arimaa player #706
Gender:
Posts: 5928
|
|
Re: 2006 WC Prediction Contest
« Reply #2 on: Nov 15th, 2005, 12:15pm » |
Quote Modify
|
on Nov 10th, 2005, 9:56pm, 99of9 wrote:| Omar I have a request for the prediction contest. I remember last time it was a little frustrating having 5% increments on the prediction choices. Especially near 0% and near 100%, 1% starts to matter a lot. I wonder if all integers between 90% and 100% could be included in the choices? |
|
I think the main reason to keep it at 5% intervals is that you get integral rewards that way. Predicting 95% wins you 99 points, while predicting 100% wins you 100 points, so we'd be getting 99.XX points on the in-between percentages.
Also, how much does 1% really matter at the extremes? Let's say the true odds on the winner are 98%. The average prediction payouts are:
100% prediction pays 92
98% prediction pays 92.16
95% prediction pays 91.8
We're talking fractions of a point on average. When you consider the sources of noise, it just doesn't matter down to the last digit.
In all likelihood the contest will go to whoever predicts best on the tossup games. The difference between a 95% prediction and a 100% prediction is almost negligible in comparison to the difference between a 45% prediction and a 55% prediction, because in the latter case one of them gets the move prediction bonus and the other doesn't.
The noise introduced by lucky guesses on coin-flip games will quite probably overwhelm any statistical nuances that one could use to eke out an extra point here and there. Frankly, that's the way I like it. The contest is more fun if gut-level intuition isn't overwhelmed by mathematics.
|
|
IP Logged |
|
|
|
99of9
Forum Guru
Gnobby's creator (player #314)
 
Gender:
Posts: 1413
|
|
Re: 2006 WC Prediction Contest
« Reply #3 on: Nov 15th, 2005, 2:52pm » |
Quote Modify
|
You're assuming that the player always *wins* the 98% games. The difference between 95% and 98% becomes quite large whenever there is an upset victory. I think predicting actual probabilities of upsets is an interesting thing in itself rather than guessing coin flip games, especially in light of all the stuff that Ryan has been working on.
|
|
IP Logged |
|
|
|
Fritzlein
Forum Guru
Arimaa player #706
Gender:
Posts: 5928
|
|
Re: 2006 WC Prediction Contest
« Reply #4 on: Nov 15th, 2005, 4:59pm » |
Quote Modify
|
on Nov 15th, 2005, 2:52pm, 99of9 wrote:| You're assuming that the player always *wins* the 98% games. |
|
No, I'm not assuming that the favorite always wins. I'm assuming that your prediction of 98% is totally accurate (i.e. there is exactly a 2% chance of upset) and calculating how much you lose on average by being forced to pick 100% or 95% instead. If you are right about your 98% prediction, the chance of big swing due to an upset is quite small. Here's the complete calculation:
100% prediction pays 92 = 0.98*100 - 0.02*300
98% prediction pays 92.16 = 0.98*99.84 - 0.02*284.16
95% prediction pays 91.8 = 0.98*99 - 0.02*261
I agree with you that predicting probabilities of upsets is very interesting. You get the highest possible payoff on average by being as accurate as possible in your percentages.
On the other hand, one might accept a small penalty in average payoff for an increased variance, on the theory that only the top three places pay out, so it is better to have a lower chance of an extremely good result than a higher chance of a merely above average result.
|
|
IP Logged |
|
|
|
99of9
Forum Guru
Gnobby's creator (player #314)
Gender:
Posts: 1413
|
|
Re: 2006 WC Prediction Contest
« Reply #5 on: Nov 15th, 2005, 6:26pm » |
Quote Modify
|
Ok sorry, I didn't read your post with enough care.
I still think it would help to have smaller increments (especially near the top). You are right that on *average* the difference will be negligible, but that is because in 98% of cases the expected winner wins, and the difference matters not at all. But when upsets happen, it matters a lot, especially given that the 1st-2nd placings in last year's prediction comp were decided by less than 30 points:
[tt]
Aamir 1042
omar 1014
mouse 942
Fritzlein 917
99of9 874
[\tt]
Quote:| On the other hand, one might accept a small penalty in average payoff for an increased variance, on the theory that only the top three places pay out, so it is better to have a lower chance of an extremely good result than a higher chance of a merely above average result. |
|
Interesting theory. My gut feeling is that humans gamble this way too much, and thus over many games, a steady rational performer will eventually come through  . (Although your theory certainly has weight if you are a few points behind and nearing the end of the WC)
|
| « Last Edit: Nov 15th, 2005, 6:27pm by 99of9 » |
IP Logged |
|
|
|
Fritzlein
Forum Guru
Arimaa player #706
Gender:
Posts: 5928
|
|
Re: 2006 WC Prediction Contest
« Reply #6 on: Nov 15th, 2005, 7:38pm » |
Quote Modify
|
on Nov 15th, 2005, 6:26pm, 99of9 wrote:| My gut feeling is that humans gamble this way too much, and thus over many games, a steady rational performer will eventually come through . |
|
Last year Aamir won by predicting 100% on every game, except that he didn't predict once or twice and took a minimum penalty for those games. This might encourage one to subscribe to the "high variance" theory.
On the other hand, last year there were a total of 11 games on which to wager. This year, barring forfeits, there will be 30 or 31. That might encourage one to play the percentages, confident that wild guessers will eventually stumble.
But as you point out, it will also depend on your standing in relation to the leaders, and on how much time remains in the contest. The nearer the contest is to the end, the more the leaders should be inclined to hedge their bets, and the less the trailers have to lose by being radical.
|
|
IP Logged |
|
|
|
Ryan_Cable
Forum Guru
Arimaa player #951
Gender:
Posts: 138
|
|
Re: 2006 WC Prediction Contest
« Reply #7 on: Nov 16th, 2005, 12:44am » |
Quote Modify
|
on Nov 15th, 2005, 2:52pm, 99of9 wrote:| I think predicting actual probabilities of upsets is an interesting thing in itself rather than guessing coin flip games, especially in light of all the stuff that Ryan has been working on. |
|
Making up an entire March Madness style prediction for a FDE tournament would be a fascinating exercise if someone comes up with a good way of scoring it. Maybe give people the opportunity to redo their predictions after each round but with a penalty for each change. Just predicting what round each player goes out in is difficult, much less predicting who they play along the way. Definitely, something to think about for next year’s prediction contest if we keep FDE.
The tournament has 30 or 31 games in it so there will be <2^31~=2*10^9 possible tournaments. My highly un-optimized python script does about 1000 tournaments per second, so a similar script that explicitly calculated the probability of every possible tournament would take 23+ days or about half the tournament. :-( However, I think a clever algorithm that makes efficient use of space and time should be able to calculate exactly the odds of each player going out in each round and exactly the odds of each player winning the game given a set of StDev 0 ratings. Coding this algorithm would be very tricky though. For a 16 player single elimination tournament, the chance that seed 1 survives round 2 is:
P(1sr2) = P(1b16) * (P(8b9) * P(1b8) + P(9b8) * P(1b9))
where P(1b16) is the probability that seed 1 beats seed 16.
The chance that seed 1 survives round 3 can be simplified to:
P(1sr3) = P(1sr2) * (P(4sr2) * P(1b4) + P(5sr2) * P(1b5) + P(12sr2) * P(1b12) + P(13sr2) * P(1b13))
The number of required calculations roughly doubles each round, and for N players the tournament has log2(N) rounds. The sum of 2^x from x = 0 to log2(N) - 1 is equal to N - 1. Thus to calculate the odds for each of the N players in the tournament takes O(N^2) calculations, which is much smaller than the 2^(N - 1) possible tournaments. Thus it would take a negligible amount of time to calculate the odds for a 128 person FIDE knockout tournament. Most importantly it should be fairly easy to implement the necessary equations in loops, rather than writing them all out by hand.
The situation for the 16 player FDE is much more complicated. In round 2, seed 1 can play any of seeds 8 through 15. If seed 1 won in round 1, then he plays the lowest ranked person who won in round 1; if seed 1 lost in round 1, he plays the lowest ranked person who lost in round 1.
After round 2 and all subsequent rounds, each seed is associated with three probabilities for the number of losses he could have. Thus the chance of seed 1 being eliminated in round looks like:
P(1L2r2) = P(16b1) * (P(2b15) * P(15b1) + P(15b2) * P(3b14) * P(14b1) + P(15b2) * P(14b3) * P(4b13) * P(13b1) + …
+ P(15b2) * P(14b3) * P(13b4) * P(12b5) * P(11b6) * P(10b7) * P(8b9) * P(9b1) + P(15b2) * P(14b3) * P(13b4) * P(12b5) * P(11b6) * P(10b7) * P(9b8) * P(8b1)
which can be simplified by factoring. The probability of seed 1 going undefeated is the mirror image of this. The probability of just 1 loss, is
P(1L1r2) = 1 - P(1L0r2) - P(1L2r2)
The probabilities for the other seeds are probably somewhat more complicated. However, there is a big problem with this approach; the requirement that people play the people they have played least. This means that the tournament is path dependent. Also you have to keep track of byes. It should be possible to handle all of this and still have the calculations run in a reasonable amount of time, but I don’t have any idea how to translate it into code that doesn’t take thousands of lines of code or more.
Unless someone sees some clever way to make this work without huge effort, I will stick with the simulations, which are already significantly more precise than the ratings are accurate. The only big advantage of a program that makes exact calculations is the ability measure the sensitivity of the tournament to small perturbations, like directly making changes to the odds for pairs of players to account for Achilles’ heels.
on Nov 15th, 2005, 4:59pm, Fritzlein wrote:| On the other hand, one might accept a small penalty in average payoff for an increased variance, on the theory that only the top three places pay out, so it is better to have a lower chance of an extremely good result than a higher chance of a merely above average result. |
|
The goal of the prediction contest is not to maximize your score per se, it is to maximize the probability of having the top score or at least within the top 3. For the 98% case, I think it is best to predict 95%. The 1 point you give up if the favorite wins will most likely vanish into the move per game noise. If that very rare upset does happen the 39 point advantage might be decisive.
It depends on how many people are in the contest, but if there are as many as there are players, then I believe slow and steady is unlikely to win the race. I think the winners will be people who go against the grain and are right at least once. Or maybe someone who is very good/lucky at estimating moves per game.
on Nov 15th, 2005, 7:38pm, Fritzlein wrote:| The nearer the contest is to the end, the more the leaders should be inclined to hedge their bets, and the less the trailers have to lose by being radical. |
|
Yes, the horizon effect is substantial. Imagine a final game for the title between 99of9 and PMertens. One could potentially make relative gains of 300+ points by predicting 100% for PMertens, and if you are 100+ points out of the money what do you have to loose?
|
|
IP Logged |
|
|
|
PMertens
Forum Guru
Arimaa player #692
Gender:
Posts: 437
|
|
Re: 2006 WC Prediction Contest
« Reply #8 on: Nov 16th, 2005, 2:59am » |
Quote Modify
|
statistically I will win against 99of9 anyway
|
|
IP Logged |
|
|
|
Fritzlein
Forum Guru
Arimaa player #706
Gender:
Posts: 5928
|
|
Re: 2006 WC Prediction Contest
« Reply #9 on: Nov 16th, 2005, 9:00am » |
Quote Modify
|
on Nov 16th, 2005, 12:44am, Ryan_Cable wrote:| Making up an entire March Madness style prediction for a FDE tournament would be a fascinating exercise if someone comes up with a good way of scoring it. |
|
Predicting the whole tourney in advance would be mind-boggling, and probably a lot of fun, but I'll bet Omar wouldn't go for it. He would probably rather force the spectators to come back each week and stay actively involved, rather than allow folks to predict and come back two months later to see how they did.
Quote:| The goal of the prediction contest is not to maximize your score per se, it is to maximize the probability of having the top score or at least within the top 3. For the 98% case, I think it is best to predict 95%. The 1 point you give up if the favorite wins will most likely vanish into the move per game noise. If that very rare upset does happen the 39 point advantage might be decisive. |
|
This cuts both ways, though. If there are ten matchups in the tourney that are 98%, then there probably won't be an upset among them, and the extra ten points from predicting 100% could be decisive too. This isn't an automatic hedging situation in my mind.
Of course, towards the beginning of the tourney you don't know whether the standings will be close or not later on, so it is really a matter of style right now. If the true winning probability is 98% I'd rather go with 100% to increase my variance if it is early in the tourney, or if it is late and I am trailing. On the other hand, if I am ranked near the top, I'd go with 95% early and even lower than 95% late.
Quote:| It depends on how many people are in the contest, but if there are as many as there are players, then I believe slow and steady is unlikely to win the race. I think the winners will be people who go against the grain and are right at least once. Or maybe someone who is very good/lucky at estimating moves per game. |
|
It's a good point that the number of contestants matters too. There were 19 contestants last year, so we might be close to 30 this year. The larger number of games this year favors slow and steady, but if there is a larger pool of competitors, surely one of them will get lucky with outrageous guessing.
Quote:| Yes, the horizon effect is substantial. Imagine a final game for the title between 99of9 and PMertens. One could potentially make relative gains of 300+ points by predicting 100% for PMertens, and if you are 100+ points out of the money what do you have to loose? |
|
Excellent example. I keep forgetting that variance is relative to the field. If everyone else is betting 100% on one side of a contest, then for me to make the "wild" 100% bet that everyone else is making provides no variance at all.
On the other hand, as I recall from last year, the field usually spread out substantially in their predictions. Even in coin flip games there would be plenty of people predicting 100% on both ends. Probably no matter what predictions you make, you'll be different from some segment of the playing pool, and have a chance to gain on them.
From my conversations with other people last year, I inferred that 99of9 and myself were the most conservative betters, hedging against upsets all the time. That stood us in good stead against most of the field, but that darned Aamir had picked Naveed to upset 99of9 (at 100%, no less!) and that gain ultimately proved unsurmountable. We got a consolation prize for 4th and 5th place last year, but if we take 4th and 5th again this year, we will walk away empty-handed.
|
|
IP Logged |
|
|
|
Adanac
Forum Guru
Arimaa player #892
Gender:
Posts: 635
|
|
Re: 2006 WC Prediction Contest
« Reply #10 on: Nov 16th, 2005, 10:58am » |
Quote Modify
|
on Nov 16th, 2005, 9:00am, Fritzlein wrote:
From my conversations with other people last year, I inferred that 99of9 and myself were the most conservative betters, ...
|
|
Are the predictions private, meaning that we only see the total scores but not the individual predictions? I was looking forward to seeing how everyone predicted each game, but it's understandable that some people might want to have hidden predictions in order to avoid offending players.
And, sorry for steering away from the main discussion, but I have a question about the scoring:
Players cannot select the number of moves on their own games. If a player wins they will get 100 points minus the number of moves it took to win. If the player loses they will get points equal to the number of moves the game took. This encourages a winning player to try and win as fast as possible and it encourages a losing player to try and prolong the game as much as possible. Although after 70 moves a losing player will not get more points for prolonging the game. This is so that a player does not get more points for losing a long (over 70 move) game than for winning. In case of a draw both players get 25 points.
So if 2 players each predict themselves with 55% likelihood, then the winner of a 100 move game gets 30 for winning + 21 prediction points (total 51) versus 70 minus 19 (total 51) for the loser. But they each get 25 points for a draw. Doesn't that mean that if the losing player had seen a potential drawing move, it's actually better not to make it? I'm going to predict all 8 first round games to end in a draw, so I want everyone to have as much incentive as possible. ;)
|
|
IP Logged |
|
|
|
Fritzlein
Forum Guru
Arimaa player #706
Gender:
Posts: 5928
|
|
Re: 2006 WC Prediction Contest
« Reply #11 on: Nov 16th, 2005, 12:54pm » |
Quote Modify
|
on Nov 16th, 2005, 10:58am, Adanac wrote:| Are the predictions private, meaning that we only see the total scores but not the individual predictions? |
|
Yep, we only see aggregate predictions, so individual predictions can only be inferred from the standings, or from what other people tell you.
Quote:| So if 2 players each predict themselves with 55% likelihood, then the winner of a 100 move game gets 30 for winning + 21 prediction points (total 51) versus 70 minus 19 (total 51) for the loser. But they each get 25 points for a draw. Doesn't that mean that if the losing player had seen a potential drawing move, it's actually better not to make it? |
|
Can we predict draws? I would advise against such predictions, because the tournament rules say draws are not allowed. Capturing all opposing rabbits is a win within the WC. Even in the esoteric situation that the game oversteps time limit and is therefore decided by score, which happens to be tied, there is no draw, because Silver wins that case.
|
|
IP Logged |
|
|
|
Ryan_Cable
Forum Guru
Arimaa player #951
Gender:
Posts: 138
|
|
Re: 2006 WC Prediction Contest
« Reply #12 on: Nov 16th, 2005, 3:19pm » |
Quote Modify
|
on Nov 16th, 2005, 12:54pm, Fritzlein wrote:| Yep, we only see aggregate predictions, so individual predictions can only be inferred from the standings, or from what other people tell you. |
|
Are the standings published after every game or just every round?
on Nov 16th, 2005, 10:58am, Adanac wrote:| So if 2 players each predict themselves with 55% likelihood, then the winner of a 100 move game gets 30 for winning + 21 prediction points (total 51) versus 70 minus 19 (total 51) for the loser. But they each get 25 points for a draw. Doesn't that mean that if the losing player had seen a potential drawing move, it's actually better not to make it? |
|
When it comes to making gameplay decisions based on the prediction contest, I think we mostly have to trust in people’s honesty. The rules prevent people from personally gaining from betting against themselves, but strong favorites can still make 300+ point relative gains by predicting 55% for themselves and throwing the game in ~70+ moves. Of course if you predict 55% for yourself against a very weak opponent, it is very strong evidence you are up to no good. Still, I think the prohibition on predicting against oneself mostly just penalizes weak underdogs, who in 2 games will probably loose ~200 relative points by betting on themselves when it is wise not to. A computer savvy cheater could just set up sock puppets anyway.
|
|
IP Logged |
|
|
|
Fritzlein
Forum Guru
Arimaa player #706
Gender:
Posts: 5928
|
|
Re: 2006 WC Prediction Contest
« Reply #13 on: Nov 16th, 2005, 5:34pm » |
Quote Modify
|
on Nov 16th, 2005, 3:19pm, Ryan_Cable wrote:| Are the standings published after every game or just every round? |
|
Just every round. Otherwise you could deduce fairly well what individuals had predicted.
Quote:| When it comes to making gameplay decisions based on the prediction contest, I think we mostly have to trust in people’s honesty. The rules prevent people from personally gaining from betting against themselves, but strong favorites can still make 300+ point relative gains by predicting 55% for themselves and throwing the game in ~70+ moves. |
|
If you are strong enough that everyone is predicting 100% on you, then throwing the game probably costs you more in expected share of the $500/$200 tournament prize pool than it gains you in expected share of the $50/$30/$20 prediction prize pool. For example Belbo has at least $15 of equity from, say, 1% chance of winning the Championship and a 5% chance of being runner up. The only game he could throw as a favorite is the first one, and it's hard to see him gaining $15 of prediction prize equity in that way. It's only one game of thirty, and in any case conservative folks like me are not predicting him anywhere near 100%, so he wouldn't gain much on us. And of course BlackKnight would gain even more than Belbo! Bigger favorites could make a bigger average gain by intentionally losing, but they would be throwing away even more tournament equity, so I don't see this ever paying off.
In a few years Arimaa will have become sufficiently popular that we can bar the participants in the World Championship from also participating in the prediction contest. In the mean time, I think you are right that we have to basically trust people to behave honsetly, but I also think Omar has done a decent job of removing blatant incentives to dishonesty.
Quote:| Still, I think the prohibition on predicting against oneself mostly just penalizes weak underdogs, who in 2 games will probably loose ~200 relative points by betting on themselves when it is wise not to. |
|
This handicap is moderated by the fact that everyone except the eventual champion will lose twice when forced to bet on himself, but it's true that the biggest underdogs stand to lose the most by this rule, because when stronger players lose, more of the field was betting on them too.
Another moderating factor is that, although betting on underdogs rarely pays off, when it does pay off, it pays off big.
A final moderating factor is that the favorites who keep winning have an offsetting handicap in not being able to predict their own game length. Looking at the six games in the semifinals and finals from last year, the predictors averaged about 75 points per game from the game length portion (when they called the winner correctly), whereas the winners of the games averaged only 52 points for speed of victory. If I survive into the late rounds, I'll be giving the field more than 20 prediction points per game I play.
Still, I admit it does work out badly for the low seeds on the balance. Here's a small idea to compensate underdogs: In recompense for being forced to bet on yourself, if you lose when the average bet was against you, you get (average bet) - 50 points for free. For example if one guy bets for you at 55% and nine folks bet against you at 95%, the average bet was 90% against you, so if you lose you get 40 free prediction points. This will never turn into an incentive to lose on purpose, it is just a modest consolation prize.
The net result of favorites not being able to predict number of moves and underdogs not being able to bet against themselves is that non-players have a slight advantage over players in predicting. This is as it should be.
Quote:| A computer savvy cheater could just set up sock puppets anyway. |
|
Tell me more about this way of cheating. I'm curious.
|
|
IP Logged |
|
|
|
99of9
Forum Guru
Gnobby's creator (player #314)
Gender:
Posts: 1413
|
|
Re: 2006 WC Prediction Contest
« Reply #14 on: Nov 16th, 2005, 6:31pm » |
Quote Modify
|
on Nov 16th, 2005, 5:34pm, Fritzlein wrote:| Tell me more about this way of cheating. I'm curious. |
|
I think he just means setting up extra accounts so you can have extra shots at the prize pool.
|
|
IP Logged |
|
|
|
|
Home
Help
Search
Members
Login
Register
Reply
Notify of replies
Send Topic
Print