Author |
Topic: Prediction scoring (Read 2194 times) |
|
99of9
Forum Guru
Gnobby's creator (player #314)
 
Gender: 
Posts: 1413
|
 |
Prediction scoring
« on: Oct 1st, 2004, 11:41am » |
 Quote  Modify
|
I have 2 issues with the prediction competition scoring method you've put up.
1) I'm not sure the "lose 100 points for losing the game" is a good idea. Why should a better player get bonuses in the prediction comp as well? I think give both the winner and the loser 100 points.
2) I love your idea of staking different amounts on different games. "I'm very sure" or "I think he might win". But as it stands, if you're going for first prize, you might as well say you're very sure on every round... and take your chances. Even if you're not going for first prize, if your expectation value is positive, then it will always be even more positive when the stakes are multiplied.
Instead I propose that you get +50 for winning a big bet, but -100 for losing a big bet, and similarly +10 for winning a small bet, but -20 for losing a small bet.
Omar objected to this latter proposal saying it just encourages you to bet small at first, but big on the final games. However I think that the earlier games are easier to predict anyway. I personally would be willing to stake quite a lot on the early games (even with this double penalty method), then at the end I am often quite unsure, so may choose a lower stake (unless I was WAY behind  ).
|
|
 IP Logged |
|
|
|
Fritzlein
Forum Guru
Arimaa player #706
Gender:
Posts: 5928
|
|
Re: Prediction scoring
« Reply #1 on: Oct 1st, 2004, 12:00pm » |
Quote Modify
|
The fundamental problem with lots of prediction contests is that there are too many obvious choices. If everyone is predicting the same way, where's the interest?
To correct this problem, we want to introduce a notion of "How sure are you?" to create distinctions between people who are otherwise picking the same winner. A simple but extremely robust rewards system comes to us from statistics, namely the squares-of-errors metric. It would go something like this:
For each game you make a prediction in terms of percentages, let's say 80-20. Merely for making a wager you get 50^2 points. Then after the game is over, you lose in proportion to how wrong you were. In the 80-20 example, if your favorite wins, you lose 20^2 points, but if your favorite loses you lose 80^2 points. Or if you like, you net 50^2-20^2 for being right, and 50^2-80^2 for being wrong.
The great feature of this system is that you have to take a risk to win, but being too risky is penalized. If you just enter a 50-50 guess you are guaranteed to break even: you can't win or lose. It's the same as not predicting. On the other hand, if you enter all your guesses at 100-0, you are risking too much. Say you predict on four games at 100-0, and were right three times. You net zero points. But if you hedge and enter four predictions at 75-25, and are right three times, then you net 2500 points.
If you are thinking about entering a bet of 100-0, consider that being right only gets you 1 point more than a 99-1 bet, but being wrong loses you 199 points more. Do you really feel the odds are that good? On the other hand, if you just bet 51-49 each time, you gain only 99 points for being right, even on the obvious games, while someone who is right on a 60-40 bet gets 900 points.
You can show mathematically that if the true percentage chance of a player winning is X, then the best average payoff in this system is obtained by predicting (X , 100-X) for that game.
I love this metric. What does everyone else think?
|
| « Last Edit: Oct 1st, 2004, 12:18pm by Fritzlein » |
IP Logged |
|
|
|
99of9
Forum Guru
Gnobby's creator (player #314)
Gender:
Posts: 1413
|
|
Re: Prediction scoring
« Reply #2 on: Oct 1st, 2004, 12:07pm » |
Quote Modify
|
I think it's pretty neat too. This, combined with "predict the number of moves" (with the same method that Omar currently suggests), would be fabulous. In fact maybe hold separate contests... game lengths, and winners?
|
|
IP Logged |
|
|
|
omar
Forum Guru
Arimaa player #2
Gender:
Posts: 1003
|
|
Re: Prediction scoring
« Reply #3 on: Oct 1st, 2004, 4:10pm » |
Quote Modify
|
I think it's good except that it highly increases the chances
of mathematicians winning
But seriously I like it. Good thing we have a mathematician on board.
Now is there a mathematically correct way to handle players predicting on their own game.
|
|
IP Logged |
|
|
|
99of9
Forum Guru
Gnobby's creator (player #314)
Gender:
Posts: 1413
|
|
Re: Prediction scoring
« Reply #4 on: Oct 1st, 2004, 4:44pm » |
Quote Modify
|
on Oct 1st, 2004, 4:10pm, omar wrote:| Now is there a mathematically correct way to handle players predicting on their own game. |
|
As long as we make the assumption that people will not throw their games in order to get ahead in the tipping contest
|
|
IP Logged |
|
|
|
99of9
Forum Guru
Gnobby's creator (player #314)
Gender:
Posts: 1413
|
|
Re: Prediction scoring
« Reply #5 on: Oct 1st, 2004, 6:57pm » |
Quote Modify
|
Great job putting in Karl's method - I think it will make things very interesting. You decided not to let us type in a number? That would allow increments smaller than 5... (but would have the disadvantage that we would need to calculate the points equation on our own).
on Oct 1st, 2004, 11:41am, 99of9 wrote:1) I'm not sure the "lose 100 points for losing the game" is a good idea. Why should a better player get bonuses in the prediction comp as well? I think give both the winner and the loser 100 points.
|
|
I still don't understand why the winner gets more tipping points than the loser on their own games. That means a good arimaa player will automatically get an advantage over a bad arimaa player in the tipping contest. This might not be so bad in a knockout tourney like the world championship, because the loser then gets to do actual tipping afterward, and might gain there. But it is bad for a round-robin, because there it is just senselessly punishing the bad players in tipping as well as their games.
|
| « Last Edit: Oct 1st, 2004, 6:59pm by 99of9 » |
IP Logged |
|
|
|
Fritzlein
Forum Guru
Arimaa player #706
Gender:
Posts: 5928
|
|
Re: Prediction scoring
« Reply #6 on: Oct 1st, 2004, 8:44pm » |
Quote Modify
|
I see 99of9's point that good players have an advantage in the prediction contest because they get points for winning games. However, I think it is of primary importance that you never have an incentive to throw a game. You should never be allowed to bet against yourself.
The way to incorporate that along with the least squares formula is simply to require a minimum bet of fifty on yourself. If you don't want to bet more on yourself, fine, but you can't bet less. If you bet only fifty, it's just like not betting: you will break even.
The good players still have an advantage, because they are allowed to bet on themselves, all the way up to a 100 percent prediction, but at least the poor players don't lose points for losing games (assuming they bet wisely).
|
|
IP Logged |
|
|
|
99of9
Forum Guru
Gnobby's creator (player #314)
Gender:
Posts: 1413
|
|
Re: Prediction scoring
« Reply #7 on: Oct 2nd, 2004, 9:49am » |
Quote Modify
|
on Oct 1st, 2004, 8:44pm, Fritzlein wrote:However, I think it is of primary importance that you never have an incentive to throw a game.
The way to incorporate that along with the least squares formula is simply to require a minimum bet of fifty on yourself. If you don't want to bet more on yourself, fine, but you can't bet less. If you bet only fifty, it's just like not betting: you will break even.
|
|
Yes, this seems less arbitrary than the hard-wired points system. It also means that someone who keeps on winning hasn't got either an unfair advantage or disadvantage against the pure spectators.
I understand it probably won't be implemented for the EU-US, but it would be nice to see in the World Championship.
I guess players still shouldn't get to bet on the number of moves in their game, no matter what. So maybe they need to be compensated a bit for that.
on Oct 1st, 2004, 8:44pm, Fritzlein wrote:| The good players still have an advantage, because they are allowed to bet on themselves, all the way up to a 100 percent prediction, but at least the poor players don't lose points for losing games (assuming they bet wisely). |
|
This is not too bad at all in a knockout tournament like the World Championships, because everyone except the eventual winner is sure to lose exactly one game. So if they're good pedictors, even of their own demise, all will only miss out once.
|
| « Last Edit: Oct 2nd, 2004, 10:03am by 99of9 » |
IP Logged |
|
|
|
Fritzlein
Forum Guru
Arimaa player #706
Gender:
Posts: 5928
|
|
Re: Prediction scoring
« Reply #8 on: Oct 2nd, 2004, 2:36pm » |
Quote Modify
|
on Oct 2nd, 2004, 9:49am, 99of9 wrote:
I guess players still shouldn't get to bet on the number of moves in their game, no matter what. So maybe they need to be compensated a bit for that.
|
|
Yes, you sholdn't be allowed to be on the length of a game you are involved in. A player involved in a game could get a fixed compensation, or they could get a sliding compensation based on how fast they win, for example 100*(25 / number of moves it takes to win), with a maximum of 100. If you get more for winning faster, then you will still have no incentive to throw the game or play badly.
|
|
IP Logged |
|
|
|
99of9
Forum Guru
Gnobby's creator (player #314)
Gender:
Posts: 1413
|
|
Re: Prediction scoring
« Reply #9 on: Oct 2nd, 2004, 3:08pm » |
Quote Modify
|
on Oct 2nd, 2004, 2:36pm, Fritzlein wrote:compensation based on how fast they win, for example 100*(25 / number of moves it takes to win), with a maximum of 100. If you get more for winning faster, then you will still have no incentive to throw the game or play badly.
|
|
Or the winner could get (100-number of moves), and the loser get (number of moves). That way all Omar's priorities would be incentivised... the losing player would have reason not to resign/lose-easily. It would be zero sum, so collaboration would be useless.
|
| « Last Edit: Oct 2nd, 2004, 3:09pm by 99of9 » |
IP Logged |
|
|
|
omar
Forum Guru
Arimaa player #2
Gender:
Posts: 1003
|
|
Re: Prediction scoring
« Reply #10 on: Oct 5th, 2004, 4:23am » |
Quote Modify
|
If we had thousands of spectators, I would not even consider allowing the players to make predictions on the games; but in the interm we have to deal with it
I will change it to allow players to only bet on themselves with a minimum of 55% and up to 100%. Also using 100 - (number of moves) for the winner and (number of moves) for the loser makes sense. Usually the predictor does not get any points from the number of moves component if the player they bet on loses, but I think in the case of the predictor and player being the same it provides an incentive for the losing player to put up a fight for as long as possible.
Does this sound OK to everyone.
|
|
IP Logged |
|
|
|
Fritzlein
Forum Guru
Arimaa player #706
Gender:
Posts: 5928
|
|
Re: Prediction scoring
« Reply #11 on: Oct 5th, 2004, 10:21am » |
Quote Modify
|
Sounds great. I would worry that the loser in a 60-move game gets more points than the winner, but being forced to bet on yourself at least 55% costs you 21 points for the loss, so this still seems fair. I don't think it will arise, but just in case maybe there should be a cutoff at 100 moves, so that the winner doesn't actually lose points for taking too many moves.
|
|
IP Logged |
|
|
|
MrBrain
Forum Guru
Arimaa player #344
Gender:
Posts: 148
|
|
Re: Prediction scoring
« Reply #12 on: Oct 5th, 2004, 2:37pm » |
Quote Modify
|
Fritzlein's original suggestion is basically the same as the function that I developed and use in my ProbabilityFootball contests. I found out later that it's something called "the quadratic scoring rule."
I'm not attempting to advertise here (I don't really need or want a lot more players), but the URL to a table for such a scoring function is at http://ProbabilityFootball.com/function.html.
|
| « Last Edit: Oct 5th, 2004, 2:41pm by MrBrain » |
IP Logged |
|
|
|
omar
Forum Guru
Arimaa player #2
Gender:
Posts: 1003
|
|
Re: Prediction scoring
« Reply #13 on: Oct 5th, 2004, 9:58pm » |
Quote Modify
|
Hey Brian; I didn't know you were also a web master. Nice site and interesting idea.
I was just about to ask Karl what is the proper way to handle draws. But looking at your table it seems that you take the average of the winning score and losing score. It is usually a negitive number, but I guess that's the matematically correct way to handle draws with this formula. I guess it kind of makes sense too; the ones who said 50% don't get punished, but the ones who confidently predicted 100% get punished the most.
For the number of move component of the score both players will get 25 points because they should normally get 50 points each, but because we divide by 2 in case of draws they each get 25 points. Does this sound reasonable?
|
|
IP Logged |
|
|
|
omar
Forum Guru
Arimaa player #2
Gender:
Posts: 1003
|
|
Re: Prediction scoring
« Reply #14 on: Oct 6th, 2004, 10:53am » |
Quote Modify
|
I was just thinking if we should make the game length prediction independent of the winner perdiction. Currently if the player you selected wins you get points for the game length based on how accurately you predicted it. If the player you selected loses then you don't get any points for the game length prediction. Maybe we should give points for the game length regardless of who wins so that it will be independent of the winner prediction.
What do you think?
|
|
IP Logged |
|
|
|
|
Home
Help
Search
Members
Login
Register
Reply
Notify of replies
Send Topic
Print