Author |
Topic: Prediction scoring (Read 2196 times) |
|
Fritzlein
Forum Guru
Arimaa player #706
Gender: 
Posts: 5928
|
 |
Re: Prediction scoring
« Reply #15 on: Oct 6th, 2004, 2:59pm » |
 Quote  Modify
|
One interesting point of linking the two predictions is that the length of the game might be correlated to the winner, e.g. we might expect that the game will be short if A wins, but long if B wins, and predict accordingly.
More importantly, however, the current linkage puts a great deal of emphasis on picking the winner correctly (at any percentage) so as to be eligible for the 100 points for picking the length correctly. If the true odds on a game are 50-50, then there is no good percentage bet, i.e. no percentage bet that will net points on average. But if picking the winner correctly makes you eligible for the length contest, then there is some value in being right, even on a pure guess. I like this feature.
In contrast, if you were to make the winner prediction and the game length prediction independent, then it would make it less like a lottery where you win based on guessing right, and more like a mathematical contest of predicting averages. There would be more skill involved that way, skill of a very mathematical nature. I'm not sure it would be good to make the prediction contest into a game of mathematical skill. I think it is more fun if lucky guesses are rewarded to some degree. Therefore I suggest keeping the current linkage.
If you want to emphasize the skill more than the luck, I would suggest (instead of delinking) that you change the relative weights of the two components, e.g. make the points available for predicting game length only half of what is available for making a percentage bet. That increases the relative importance of the correct percentage prediction as opposed to merely the correct winner prediction.
Just some thoughts...
|
| « Last Edit: Oct 6th, 2004, 3:00pm by Fritzlein » |
 IP Logged |
|
|
|
99of9
Forum Guru
Gnobby's creator (player #314)
 
Gender:
Posts: 1413
|
|
Re: Prediction scoring
« Reply #16 on: Oct 7th, 2004, 7:55pm » |
Quote Modify
|
on Oct 5th, 2004, 9:58pm, omar wrote:
For the number of move component of the score both players will get 25 points because they should normally get 50 points each, but because we divide by 2 in case of draws they each get 25 points. Does this sound reasonable?
|
|
No, you shouldn't divide by 2 in this case. If a player wins, by my (100-N) method, the players scores for this component both always add to 100. This should still happen for draws. 50 each.
|
|
IP Logged |
|
|
|
MrBrain
Forum Guru
Arimaa player #344
Gender:
Posts: 148
|
|
Re: Prediction scoring
« Reply #17 on: Oct 8th, 2004, 9:17am » |
Quote Modify
|
I like Omar's idea of making the winner picks and the game-length picks independent. Scaling and scoring should be balanced between the two to make them approximately equal in importance. For example, we could try the following possible scheme:
Use the quadratic scoring rule for the winner predictions. Use the same scaling as in the link I gave previously. (An advantage is that if someone comes into the contest late, they might not be too far behind someone that started earlier, since the neutral pick of 50-50 gives 0 points, the same as not picking.)
Now for the game-length prediction, take the absolute value of the difference of the prediction and the game length, multiply by 5, and subtract from 100. (People who miss the pick just get 0 points, regardless of game length.) This gives a maximum of 100 points gained, just like the winner-prediction part. But you can also lose points if you're off by more than 20 moves.
The second part could be tweaked to give it more or less emphasis.
|
|
IP Logged |
|
|
|
99of9
Forum Guru
Gnobby's creator (player #314)
Gender:
Posts: 1413
|
|
Re: Prediction scoring
« Reply #18 on: Oct 9th, 2004, 10:44am » |
Quote Modify
|
Omar, I just stopped in to see how your game against Belbo went... and noticed that the predictions were still open on it. I know you have a mechanism to close predictions before the game starts - but it's evidently not working right at the moment.
|
|
IP Logged |
|
|
|
clauchau
Forum Guru
bot Quantum Leapfrog's father
Gender:
Posts: 145
|
|
Re: Prediction scoring
« Reply #19 on: Apr 18th, 2006, 3:01pm » |
Quote Modify
|
on Oct 1st, 2004, 12:00pm, Fritzlein wrote:A simple but extremely robust rewards system comes to us from statistics, namely the squares-of-errors metric. It would go something like this:
For each game you make a prediction in terms of percentages, let's say 80-20. Merely for making a wager you get 50^2 points. Then after the game is over, you lose in proportion to how wrong you were. In the 80-20 example, if your favorite wins, you lose 20^2 points, but if your favorite loses you lose 80^2 points. Or if you like, you net 50^2-20^2 for being right, and 50^2-80^2 for being wrong.
The great feature of this system is that you have to take a risk to win, but being too risky is penalized. If you just enter a 50-50 guess you are guaranteed to break even: you can't win or lose. It's the same as not predicting. On the other hand, if you enter all your guesses at 100-0, you are risking too much. Say you predict on four games at 100-0, and were right three times. You net zero points. But if you hedge and enter four predictions at 75-25, and are right three times, then you net 2500 points.
If you are thinking about entering a bet of 100-0, consider that being right only gets you 1 point more than a 99-1 bet, but being wrong loses you 199 points more. Do you really feel the odds are that good? On the other hand, if you just bet 51-49 each time, you gain only 99 points for being right, even on the obvious games, while someone who is right on a 60-40 bet gets 900 points.
You can show mathematically that if the true percentage chance of a player winning is X, then the best average payoff in this system is obtained by predicting (X , 100-X) for that game.
I love this metric. What does everyone else think?
|
|
I've just read about another metric with the same features mentioned above plus the feature of being somehow fine with combined bets. It doesn't matter much I guess, except that it feels even more natural to my mathematical instincts than the square metric here and may result in other nice features : http://yudkowsky.net/bayes/technical.html
It's simply about multiplying your score with the prediction percentage you put on the actual winner and dividing it by 50. Or equivalently, if y is the difference between your two prediction percentages (which sum to 100), your score increases or decreases by y%.
So everytime you bet 50-50, your score doesn't change. In other words, it increases or decreases by 0%. Everytime you bet 80-20 your score is multiplied by 1.6 if the player you expected most wins, or by 0.4 if he loses. In other words, your score increases or decreases by 60% (80-20=60). Everytime you bet 100-0, your score double if you were right or gets forever zeroed if your were wrong. In other words, it increases or decreases by 100%. Most risky! Your score starts with 1 or some larger amount to avoid decimal parts.
If the numbers are getting too big, we may replace the multiplying factor by the square root of it or the tenth root of it, but huge numbers are fine with me compared to decimal parts.
We may also consider the log of this all so that scores are additive instead of multiplicative. The two really amounts to the same as far as mathematical features and ordering the betting players according to their skills are concerned.
|
| « Last Edit: Apr 20th, 2006, 1:34am by clauchau » |
IP Logged |
|
|
|
Fritzlein
Forum Guru
Arimaa player #706
Gender:
Posts: 5928
|
|
Re: Prediction scoring
« Reply #20 on: Apr 18th, 2006, 5:47pm » |
Quote Modify
|
on Apr 18th, 2006, 3:01pm, clauchau wrote:| I've just read about another metric with the same features mentioned above plus the feature of being somehow fine with combined bets. It doesn't matter much I guess, except that it feels even more natural to my mathematical instincts than the square metric here and may result in other nice features : http://yudkowsky.net/bayes/technical.html |
|
Thanks for the link, clauchau. It's an entertaining read. I agree with the essential property of both scoring systems, namely that if X% is the true probability, your best payoff in the long run is going to be betting X%. To that extent, they seem equal.
However, the feature of being allowed to bet on combined outcomes only makes a difference if the outcomes are dependent on each other, right? It allows you to look for extra patterns rather than taking each event as it comes. That can only be a feature if there are extra patterns and dependencies to be found.
We all believe that game outcomes are independent, right? For example, if I bet 70% on 99of9 beating PMertens and 60% on Adanac beating Robinson, then my joint bets on the two winners would be
42% 99of9 and Adanac
28% 99of9 and Robinson
18% PMertens and Adanac
12% PMertens and Robinson
Yes, there are other possible ways to distribute the joint probability of 70% on 99of9, 60% on Adanac, such as
30% 99of9 and Adanac
40% 99of9 and Robinson
30% PMertens and Adanac
00% PMertens and Robinson
but that would just be dumb, right? It would be like saying, "99of9 is 70% likely to win, and if he does win then Robinson is 4/7 likely to beat Adanac in the second game, but if PMertens wins the first game, then Adanac is certain to win the second game." (Or you could reverse time (reverse the causality) and say that if Robinson wins, PMertens will be demoralized and automatically lose, but if Adanac wins then PMertens will fight to the death and have a 50% of beating 99of9.  )
There doesn't seem to be a great virtue in a scoring system which allows us to look for patterns that aren't there. We would just be adding a feature which, on average, punishes anyone who makes use of it!
Or am I missing the point of the proposed scoring?
|
|
IP Logged |
|
|
|
clauchau
Forum Guru
bot Quantum Leapfrog's father
Gender:
Posts: 145
|
|
Re: Prediction scoring
« Reply #21 on: Apr 19th, 2006, 6:31am » |
Quote Modify
|
Fritzlein, you are right as far as we bet on games in the same well-defined round, as opposed to yet-undetermined future rounds... And I think that's the way we'll always bet, so I agree there is no need of change in that respect. 
Plus, I appreciate the integer arithmetics involved with the square metrics and the reasonable range of numbers involved.
In the other hand, I wasn't really hooked onto the exact justification for the metric I mentioned, except for the very fact that it is standing out among the plenty metrics available. In a probabilistic sense, as opposed to standing out in real physics, which is how the square metric feels to me now.
The resulting score for this probabilistic metric actually seems to be something like how many players would need to bet that way to be at least as much right if the games were all fair coin tosses. Or if we divide the score by the initial score and 2 to the power of the number of bets, it looks like the corresponding probability. At first glance.
I also like it that it extends the exciting "double or lose everything".
By the way, Eliezer Yudkowsky's text seems to be motivated by his (quite ambitious) research in Artificial Intelligence.
|
|
IP Logged |
|
|
|
clauchau
Forum Guru
bot Quantum Leapfrog's father
Gender:
Posts: 145
|
|
Re: Prediction scoring
« Reply #22 on: Apr 21st, 2006, 2:09pm » |
Quote Modify
|
I'm still contemplating the possible differences between the square metric and the double metric. A striking one is -- on average you are better with a 100%-0% bet than a 50%-50% bet when your favorite player's actual winning frequency is 80% -- with the square metric but not with the double metric.
Another way to see it is with the following table.
| A player's actual winning frequency | Percentage bet on that player that scores nothing with the square metric on average | Percentage bet on that player that scores nothing with the double metric on average | | 50% | 50% | 50.0% | | 60% | 70% | 69.5% | | 70% | 90% | 85.8% | | 75% | 100% | 92.0% | | 80% | | 96.4% | | 90% | | 99.9% |
|
| « Last Edit: Apr 21st, 2006, 2:13pm by clauchau » |
IP Logged |
|
|
|
Fritzlein
Forum Guru
Arimaa player #706
Gender:
Posts: 5928
|
|
Re: Prediction scoring
« Reply #23 on: Apr 21st, 2006, 2:42pm » |
Quote Modify
|
I see what you are saying, Clauchau. The square error metric punishes you in proportion to the number of percentage points you are off by. If the true percentage is 80%, then predictions of 60% and 100% are equally bad/good, because both are off the true by 20%. In the multiplicative metric, a bet of 60% is as bad/good as a bet of 93.1%. The bet of 100% is absolutely awful, because it will permanently wipe you out to lose that bet even one time.
Therefore I'll grant you a feature of your metric: it punishes absolute certainty more appropriately. Your metric would help train people to never say, "I'm absolutely positive," or "There's no way I could be wrong," or any such rot. Indeed, if Omar changed to this metric, he would probably want to eliminate the option of betting 100% so as to prevent people from eliminating themselves.
|
| « Last Edit: Apr 22nd, 2006, 7:42pm by Fritzlein » |
IP Logged |
|
|
|
clauchau
Forum Guru
bot Quantum Leapfrog's father
Gender:
Posts: 145
|
|
Re: Prediction scoring
« Reply #24 on: Apr 22nd, 2006, 3:58pm » |
Quote Modify
|
Fritzlein, I love many of your posts, they are friendly and well-thought, and this one is no exception. I hope mine weren't too unpleasant for you, as I tend to suggest something is wrong and I may know better while I should really be less certain.
|
|
IP Logged |
|
|
|
Fritzlein
Forum Guru
Arimaa player #706
Gender:
Posts: 5928
|
|
Re: Prediction scoring
« Reply #25 on: Apr 22nd, 2006, 8:13pm » |
Quote Modify
|
You are too kind, clauchau. I am too often overfond of my own ideas and dismissive of others. I too often look for evidence to support what I wish to be true, rather than accepting whatever truth the evidence indicates.
In this context, I particularly appreciate your linking to Yudkoswky's article. I would live my life better if I kept in mind the tenets of Bayesianity.
|
|
IP Logged |
|
|
|
|
Home
Help
Search
Members
Login
Register
Reply
Notify of replies
Send Topic
Print