Arimaa Forum - Print Page


    
      
        Arimaa Forum
        (http://arimaa.com/arimaa/forum/cgi/YaBB.cgi)
      

        Arimaa >> Events >> Prediction scoring
        
(Message started by: 99of9 on Oct 1^st, 2004, 11:41am)

Title: Prediction scoring
Post by 99of9 on Oct 1^st, 2004, 11:41am

I have 2 issues with the prediction competition scoring method you've put up.

1) I'm not sure the "lose 100 points for losing the game" is a good idea. Why should a better player get bonuses in the prediction comp as well? I think give both the winner and the loser 100 points.

2) I love your idea of staking different amounts on different games. "I'm very sure" or "I think he might win". But as it stands, if you're going for first prize, you might as well say you're very sure on every round... and take your chances. Even if you're not going for first prize, if your expectation value is positive, then it will always be even more positive when the stakes are multiplied.

Instead I propose that you get +50 for winning a big bet, but -100 for losing a big bet, and similarly +10 for winning a small bet, but -20 for losing a small bet.

Omar objected to this latter proposal saying it just encourages you to bet small at first, but big on the final games. However I think that the earlier games are easier to predict anyway. I personally would be willing to stake quite a lot on the early games (even with this double penalty method), then at the end I am often quite unsure, so may choose a lower stake (unless I was WAY behind :-) ).

Title: Re: Prediction scoring
Post by Fritzlein on Oct 1^st, 2004, 12:00pm

The fundamental problem with lots of prediction contests is that there are too many obvious choices. If everyone is predicting the same way, where's the interest?

To correct this problem, we want to introduce a notion of "How sure are you?" to create distinctions between people who are otherwise picking the same winner. A simple but extremely robust rewards system comes to us from statistics, namely the squares-of-errors metric. It would go something like this:

For each game you make a prediction in terms of percentages, let's say 80-20. Merely for making a wager you get 50^2 points. Then after the game is over, you lose in proportion to how wrong you were. In the 80-20 example, if your favorite wins, you lose 20^2 points, but if your favorite loses you lose 80^2 points. Or if you like, you net 50^2-20^2 for being right, and 50^2-80^2 for being wrong.

The great feature of this system is that you have to take a risk to win, but being too risky is penalized. If you just enter a 50-50 guess you are guaranteed to break even: you can't win or lose. It's the same as not predicting. On the other hand, if you enter all your guesses at 100-0, you are risking too much. Say you predict on four games at 100-0, and were right three times. You net zero points. But if you hedge and enter four predictions at 75-25, and are right three times, then you net 2500 points.

If you are thinking about entering a bet of 100-0, consider that being right only gets you 1 point more than a 99-1 bet, but being wrong loses you 199 points more. Do you really feel the odds are that good? On the other hand, if you just bet 51-49 each time, you gain only 99 points for being right, even on the obvious games, while someone who is right on a 60-40 bet gets 900 points.

You can show mathematically that if the true percentage chance of a player winning is X, then the best average payoff in this system is obtained by predicting (X , 100-X) for that game.

I love this metric. What does everyone else think?

Title: Re: Prediction scoring
Post by 99of9 on Oct 1^st, 2004, 12:07pm

I think it's pretty neat too. This, combined with "predict the number of moves" (with the same method that Omar currently suggests), would be fabulous. In fact maybe hold separate contests... game lengths, and winners?

Title: Re: Prediction scoring
Post by omar on Oct 1^st, 2004, 4:10pm

I think it's good except that it highly increases the chances
of mathematicians winning :-)

But seriously I like it. Good thing we have a mathematician on board.

Now is there a mathematically correct way to handle players predicting on their own game.

Title: Re: Prediction scoring
Post by 99of9 on Oct 1^st, 2004, 4:44pm

on 10/01/04 at 16:10:08, omar wrote:

Now is there a mathematically correct way to handle players predicting on their own game.

As long as we make the assumption that people will not throw their games in order to get ahead in the tipping contest :)

Title: Re: Prediction scoring
Post by 99of9 on Oct 1^st, 2004, 6:57pm

Great job putting in Karl's method - I think it will make things very interesting. You decided not to let us type in a number? That would allow increments smaller than 5... (but would have the disadvantage that we would need to calculate the points equation on our own).

on 10/01/04 at 11:41:49, 99of9 wrote:

1) I'm not sure the "lose 100 points for losing the game" is a good idea. Why should a better player get bonuses in the prediction comp as well? I think give both the winner and the loser 100 points.

I still don't understand why the winner gets more tipping points than the loser on their own games. That means a good arimaa player will automatically get an advantage over a bad arimaa player in the tipping contest. This might not be so bad in a knockout tourney like the world championship, because the loser then gets to do actual tipping afterward, and might gain there. But it is bad for a round-robin, because there it is just senselessly punishing the bad players in tipping as well as their games.

Title: Re: Prediction scoring
Post by Fritzlein on Oct 1^st, 2004, 8:44pm

I see 99of9's point that good players have an advantage in the prediction contest because they get points for winning games. However, I think it is of primary importance that you never have an incentive to throw a game. You should never be allowed to bet against yourself.

The way to incorporate that along with the least squares formula is simply to require a minimum bet of fifty on yourself. If you don't want to bet more on yourself, fine, but you can't bet less. If you bet only fifty, it's just like not betting: you will break even.

The good players still have an advantage, because they are allowed to bet on themselves, all the way up to a 100 percent prediction, but at least the poor players don't lose points for losing games (assuming they bet wisely).

Title: Re: Prediction scoring
Post by 99of9 on Oct 2^nd, 2004, 9:49am

on 10/01/04 at 20:44:30, Fritzlein wrote:

However, I think it is of primary importance that you never have an incentive to throw a game.

The way to incorporate that along with the least squares formula is simply to require a minimum bet of fifty on yourself. If you don't want to bet more on yourself, fine, but you can't bet less. If you bet only fifty, it's just like not betting: you will break even.

Yes, this seems less arbitrary than the hard-wired points system. It also means that someone who keeps on winning hasn't got either an unfair advantage or disadvantage against the pure spectators.

I understand it probably won't be implemented for the EU-US, but it would be nice to see in the World Championship.

I guess players still shouldn't get to bet on the number of moves in their game, no matter what. So maybe they need to be compensated a bit for that.

on 10/01/04 at 20:44:30, Fritzlein wrote:

The good players still have an advantage, because they are allowed to bet on themselves, all the way up to a 100 percent prediction, but at least the poor players don't lose points for losing games (assuming they bet wisely).

This is not too bad at all in a knockout tournament like the World Championships, because everyone except the eventual winner is sure to lose exactly one game. So if they're good pedictors, even of their own demise, all will only miss out once.

Title: Re: Prediction scoring
Post by Fritzlein on Oct 2^nd, 2004, 2:36pm

on 10/02/04 at 09:49:13, 99of9 wrote:

I guess players still shouldn't get to bet on the number of moves in their game, no matter what. So maybe they need to be compensated a bit for that.

Yes, you sholdn't be allowed to be on the length of a game you are involved in. A player involved in a game could get a fixed compensation, or they could get a sliding compensation based on how fast they win, for example 100*(25 / number of moves it takes to win), with a maximum of 100. If you get more for winning faster, then you will still have no incentive to throw the game or play badly.

Title: Re: Prediction scoring
Post by 99of9 on Oct 2^nd, 2004, 3:08pm

on 10/02/04 at 14:36:17, Fritzlein wrote:

compensation based on how fast they win, for example 100*(25 / number of moves it takes to win), with a maximum of 100. If you get more for winning faster, then you will still have no incentive to throw the game or play badly.

Or the winner could get (100-number of moves), and the loser get (number of moves). That way all Omar's priorities would be incentivised... the losing player would have reason not to resign/lose-easily. It would be zero sum, so collaboration would be useless.

Title: Re: Prediction scoring
Post by omar on Oct 5^th, 2004, 4:23am

If we had thousands of spectators, I would not even consider allowing the players to make predictions on the games; but in the interm we have to deal with it :-)

I will change it to allow players to only bet on themselves with a minimum of 55% and up to 100%. Also using 100 - (number of moves) for the winner and (number of moves) for the loser makes sense. Usually the predictor does not get any points from the number of moves component if the player they bet on loses, but I think in the case of the predictor and player being the same it provides an incentive for the losing player to put up a fight for as long as possible.

Does this sound OK to everyone.

Title: Re: Prediction scoring
Post by Fritzlein on Oct 5^th, 2004, 10:21am

Sounds great. I would worry that the loser in a 60-move game gets more points than the winner, but being forced to bet on yourself at least 55% costs you 21 points for the loss, so this still seems fair. I don't think it will arise, but just in case maybe there should be a cutoff at 100 moves, so that the winner doesn't actually lose points for taking too many moves.

Title: Re: Prediction scoring
Post by MrBrain on Oct 5^th, 2004, 2:37pm

Fritzlein's original suggestion is basically the same as the function that I developed and use in my ProbabilityFootball contests. I found out later that it's something called "the quadratic scoring rule."

I'm not attempting to advertise here (I don't really need or want a lot more players), but the URL to a table for such a scoring function is at http://ProbabilityFootball.com/function.html.

Title: Re: Prediction scoring
Post by omar on Oct 5^th, 2004, 9:58pm

Hey Brian; I didn't know you were also a web master. Nice site and interesting idea.

I was just about to ask Karl what is the proper way to handle draws. But looking at your table it seems that you take the average of the winning score and losing score. It is usually a negitive number, but I guess that's the matematically correct way to handle draws with this formula. I guess it kind of makes sense too; the ones who said 50% don't get punished, but the ones who confidently predicted 100% get punished the most.

For the number of move component of the score both players will get 25 points because they should normally get 50 points each, but because we divide by 2 in case of draws they each get 25 points. Does this sound reasonable?

Title: Re: Prediction scoring
Post by omar on Oct 6^th, 2004, 10:53am

I was just thinking if we should make the game length prediction independent of the winner perdiction. Currently if the player you selected wins you get points for the game length based on how accurately you predicted it. If the player you selected loses then you don't get any points for the game length prediction. Maybe we should give points for the game length regardless of who wins so that it will be independent of the winner prediction.

What do you think?

Title: Re: Prediction scoring
Post by Fritzlein on Oct 6^th, 2004, 2:59pm

One interesting point of linking the two predictions is that the length of the game might be correlated to the winner, e.g. we might expect that the game will be short if A wins, but long if B wins, and predict accordingly.

More importantly, however, the current linkage puts a great deal of emphasis on picking the winner correctly (at any percentage) so as to be eligible for the 100 points for picking the length correctly. If the true odds on a game are 50-50, then there is no good percentage bet, i.e. no percentage bet that will net points on average. But if picking the winner correctly makes you eligible for the length contest, then there is some value in being right, even on a pure guess. I like this feature.

In contrast, if you were to make the winner prediction and the game length prediction independent, then it would make it less like a lottery where you win based on guessing right, and more like a mathematical contest of predicting averages. There would be more skill involved that way, skill of a very mathematical nature. I'm not sure it would be good to make the prediction contest into a game of mathematical skill. I think it is more fun if lucky guesses are rewarded to some degree. Therefore I suggest keeping the current linkage.

If you want to emphasize the skill more than the luck, I would suggest (instead of delinking) that you change the relative weights of the two components, e.g. make the points available for predicting game length only half of what is available for making a percentage bet. That increases the relative importance of the correct percentage prediction as opposed to merely the correct winner prediction.

Just some thoughts...

Title: Re: Prediction scoring
Post by 99of9 on Oct 7^th, 2004, 7:55pm

on 10/05/04 at 21:58:41, omar wrote:

For the number of move component of the score both players will get 25 points because they should normally get 50 points each, but because we divide by 2 in case of draws they each get 25 points. Does this sound reasonable?

No, you shouldn't divide by 2 in this case. If a player wins, by my (100-N) method, the players scores for this component both always add to 100. This should still happen for draws. 50 each.

Title: Re: Prediction scoring
Post by MrBrain on Oct 8^th, 2004, 9:17am

I like Omar's idea of making the winner picks and the game-length picks independent. Scaling and scoring should be balanced between the two to make them approximately equal in importance. For example, we could try the following possible scheme:

Use the quadratic scoring rule for the winner predictions. Use the same scaling as in the link I gave previously. (An advantage is that if someone comes into the contest late, they might not be too far behind someone that started earlier, since the neutral pick of 50-50 gives 0 points, the same as not picking.)

Now for the game-length prediction, take the absolute value of the difference of the prediction and the game length, multiply by 5, and subtract from 100. (People who miss the pick just get 0 points, regardless of game length.) This gives a maximum of 100 points gained, just like the winner-prediction part. But you can also lose points if you're off by more than 20 moves.

The second part could be tweaked to give it more or less emphasis.

Title: Re: Prediction scoring
Post by 99of9 on Oct 9^th, 2004, 10:44am

Omar, I just stopped in to see how your game against Belbo went... and noticed that the predictions were still open on it. I know you have a mechanism to close predictions before the game starts - but it's evidently not working right at the moment.

Title: Re: Prediction scoring
Post by clauchau on Apr 18^th, 2006, 3:01pm

on 10/01/04 at 12:00:40, Fritzlein wrote:

A simple but extremely robust rewards system comes to us from statistics, namely the squares-of-errors metric. It would go something like this:

For each game you make a prediction in terms of percentages, let's say 80-20. Merely for making a wager you get 50^2 points. Then after the game is over, you lose in proportion to how wrong you were. In the 80-20 example, if your favorite wins, you lose 20^2 points, but if your favorite loses you lose 80^2 points. Or if you like, you net 50^2-20^2 for being right, and 50^2-80^2 for being wrong.

The great feature of this system is that you have to take a risk to win, but being too risky is penalized. If you just enter a 50-50 guess you are guaranteed to break even: you can't win or lose. It's the same as not predicting. On the other hand, if you enter all your guesses at 100-0, you are risking too much. Say you predict on four games at 100-0, and were right three times. You net zero points. But if you hedge and enter four predictions at 75-25, and are right three times, then you net 2500 points.

If you are thinking about entering a bet of 100-0, consider that being right only gets you 1 point more than a 99-1 bet, but being wrong loses you 199 points more. Do you really feel the odds are that good? On the other hand, if you just bet 51-49 each time, you gain only 99 points for being right, even on the obvious games, while someone who is right on a 60-40 bet gets 900 points.

You can show mathematically that if the true percentage chance of a player winning is X, then the best average payoff in this system is obtained by predicting (X , 100-X) for that game.

I love this metric. What does everyone else think?

I've just read about another metric with the same features mentioned above plus the feature of being somehow fine with combined bets. It doesn't matter much I guess, except that it feels even more natural to my mathematical instincts than the square metric here and may result in other nice features : http://yudkowsky.net/bayes/technical.html

It's simply about multiplying your score with the prediction percentage you put on the actual winner and dividing it by 50. Or equivalently, if y is the difference between your two prediction percentages (which sum to 100), your score increases or decreases by y%.

So everytime you bet 50-50, your score doesn't change. In other words, it increases or decreases by 0%. Everytime you bet 80-20 your score is multiplied by 1.6 if the player you expected most wins, or by 0.4 if he loses. In other words, your score increases or decreases by 60% (80-20=60). Everytime you bet 100-0, your score double if you were right or gets forever zeroed if your were wrong. In other words, it increases or decreases by 100%. Most risky! Your score starts with 1 or some larger amount to avoid decimal parts.

If the numbers are getting too big, we may replace the multiplying factor by the square root of it or the tenth root of it, but huge numbers are fine with me compared to decimal parts.

We may also consider the log of this all so that scores are additive instead of multiplicative. The two really amounts to the same as far as mathematical features and ordering the betting players according to their skills are concerned.

Title: Re: Prediction scoring
Post by Fritzlein on Apr 18^th, 2006, 5:47pm

on 04/18/06 at 15:01:42, clauchau wrote:

Thanks for the link, clauchau. It's an entertaining read. I agree with the essential property of both scoring systems, namely that if X% is the true probability, your best payoff in the long run is going to be betting X%. To that extent, they seem equal.

However, the feature of being allowed to bet on combined outcomes only makes a difference if the outcomes are dependent on each other, right? It allows you to look for extra patterns rather than taking each event as it comes. That can only be a feature if there are extra patterns and dependencies to be found.

We all believe that game outcomes are independent, right? For example, if I bet 70% on 99of9 beating PMertens and 60% on Adanac beating Robinson, then my joint bets on the two winners would be

42% 99of9 and Adanac
28% 99of9 and Robinson
18% PMertens and Adanac
12% PMertens and Robinson

Yes, there are other possible ways to distribute the joint probability of 70% on 99of9, 60% on Adanac, such as

30% 99of9 and Adanac
40% 99of9 and Robinson
30% PMertens and Adanac
00% PMertens and Robinson

but that would just be dumb, right? It would be like saying, "99of9 is 70% likely to win, and if he does win then Robinson is 4/7 likely to beat Adanac in the second game, but if PMertens wins the first game, then Adanac is certain to win the second game." (Or you could reverse time (reverse the causality) and say that if Robinson wins, PMertens will be demoralized and automatically lose, but if Adanac wins then PMertens will fight to the death and have a 50% of beating 99of9. :-))

There doesn't seem to be a great virtue in a scoring system which allows us to look for patterns that aren't there. We would just be adding a feature which, on average, punishes anyone who makes use of it!

Or am I missing the point of the proposed scoring?

Title: Re: Prediction scoring
Post by clauchau on Apr 19^th, 2006, 6:31am

Fritzlein, you are right as far as we bet on games in the same well-defined round, as opposed to yet-undetermined future rounds... And I think that's the way we'll always bet, so I agree there is no need of change in that respect. ;D

Plus, I appreciate the integer arithmetics involved with the square metrics and the reasonable range of numbers involved.

In the other hand, I wasn't really hooked onto the exact justification for the metric I mentioned, except for the very fact that it is standing out among the plenty metrics available. In a probabilistic sense, as opposed to standing out in real physics, which is how the square metric feels to me now.

The resulting score for this probabilistic metric actually seems to be something like how many players would need to bet that way to be at least as much right if the games were all fair coin tosses. Or if we divide the score by the initial score and 2 to the power of the number of bets, it looks like the corresponding probability. At first glance.

I also like it that it extends the exciting "double or lose everything".

By the way, Eliezer Yudkowsky's text seems to be motivated by his (quite ambitious) research in Artificial Intelligence.

Title: Re: Prediction scoring
Post by clauchau on Apr 21^st, 2006, 2:09pm

I'm still contemplating the possible differences between the square metric and the double metric. A striking one is -- on average you are better with a 100%-0% bet than a 50%-50% bet when your favorite player's actual winning frequency is 80% -- with the square metric but not with the double metric.

Another way to see it is with the following table.

A player's actual winning frequency	Percentage bet on that player that scores nothing with the square metric on average	Percentage bet on that player that scores nothing with the double metric on average
50%	50%	50.0%
60%	70%	69.5%
70%	90%	85.8%
75%	100%	92.0%
80%		96.4%
90%		99.9%

Title: Re: Prediction scoring
Post by Fritzlein on Apr 21^st, 2006, 2:42pm

I see what you are saying, Clauchau. The square error metric punishes you in proportion to the number of percentage points you are off by. If the true percentage is 80%, then predictions of 60% and 100% are equally bad/good, because both are off the true by 20%. In the multiplicative metric, a bet of 60% is as bad/good as a bet of 93.1%. The bet of 100% is absolutely awful, because it will permanently wipe you out to lose that bet even one time.

Therefore I'll grant you a feature of your metric: it punishes absolute certainty more appropriately. Your metric would help train people to never say, "I'm absolutely positive," or "There's no way I could be wrong," or any such rot. :-) Indeed, if Omar changed to this metric, he would probably want to eliminate the option of betting 100% so as to prevent people from eliminating themselves.

Title: Re: Prediction scoring
Post by clauchau on Apr 22^nd, 2006, 3:58pm

:) Fritzlein, I love many of your posts, they are friendly and well-thought, and this one is no exception. I hope mine weren't too unpleasant for you, as I tend to suggest something is wrong and I may know better while I should really be less certain.

Title: Re: Prediction scoring
Post by Fritzlein on Apr 22^nd, 2006, 8:13pm

You are too kind, clauchau. I am too often overfond of my own ideas and dismissive of others. I too often look for evidence to support what I wish to be true, rather than accepting whatever truth the evidence indicates.

In this context, I particularly appreciate your linking to Yudkoswky's article. I would live my life better if I kept in mind the tenets of Bayesianity. :-)