Author |
Topic: Replace the Scoring Function (Read 8129 times) |
|
browni3141
Forum Guru
Arimaa player #7014
Gender: 
Posts: 384
|
 |
Replace the Scoring Function
« on: Apr 19th, 2013, 3:13am » |
 Quote  Modify
|
I have created this thread because I want to discuss replacing the scoring function, because I find it a very distasteful part of the Arimaa rules.
" I suggest nothing since in my opinion there´s no need to open such discussion. So far, the score function wasn´t an issue. In a game between humans it may almost never occur that this function will decide a game. That´s the most important point."-odin73 from http://arimaa.com/arimaa/gameroom/comments.cgi?gid=261898
I think this is the opinion held by a lot of you. It has never decided an important game, so it is not a problem. I completely disagree. I think it would be harmful to the game of Arimaa (to what degree depends on the importance of the game being played) if a game ever ends on score. Think of it as car insurance. Most of us are not expecting to need it anytime soon, but we want to be prepared for the possibility of a crash.
Also, I believe that odin is wrong in saying that the scoring function has never been an issue in the past. Remember that it is not only games that end on score that are an issue; it is any game where score enters the players thoughts! I think most can agree that ideally score should play no role in the game whatsoever. In this case score has been a problem in the past. The games could be hard to find, but I recall at least two in the past year or so (one of them my own other than the joke game)
So now I'd like to compare options, some of which are old and some my own ideas. They are ordered in what I believe is worst to best. I already have an opinion on which one is best, and I'm amazed it hasn't been suggested before, at least as far as I know.
1. The current scoring function
Pro:
- The game will end in a known amount of time
- No site changes required
Con:
- It is going to be wrong much of the time
- Silver is arbitrarily chosen to win the tie-breaker
- Players may be encouraged to play ludicrously if they are getting close to the game ending on score
2. Use a material evaluator such as DAPE
Pro:
- It will be more accurate than score
Con:
- It pretty much has all the same problems as score
3. The players each get to use whatever time they have remaining to make the rest of their moves
Pro:
- The game will probably not last much longer than if score were simply used
- The proper winner is much more likely to be produced.
- The game is more likely to end naturally
- This is fairly simple
Con:
- Sudden death time controls are icky for Arimaa because it is very hard to move quickly. This is a big problem IMO.
4. Accelerating time controls
Pro:
- The proper winner is much more likely to be produced.
- There is a much greater chance for the game to end naturally.
Con:
- Seems kind of complicated
- The game is not guaranteed to end using this method only
And now for my (and in my opinion, the best) solution!
Use a human adjudicator (or panel of adjudicators)!
Pro:
- It will be the most accurate of all for determining the winner (accelerating time controls arguably being an exception)
- All omar has to do is say yes. No immediate site changes are required as long as the score result can be overridden.
- I volunteer to be one of the adjudicators, so lack of volunteers is no issue
- The game ends in a known amount of time
- The play of the players shouldn't be influenced near the end of the game like it would be for other alternatives
Con:
- A human might be biased (this chance can be mitigated with further methods)
Note that none of these really solves the problem of "drawn" positions. I would like to discuss that also, but that should probably be left to another thread, or at least until we talk about the main topic first.
I see no reason right now not to use a human adjudicator. My "Con" is practically meaningless considering how massively superior this is to score.
Discuss!
|
|
 IP Logged |
|
|
|
Boo
Forum Guru
Arimaa player #6466
Gender:
Posts: 118
|
|
Re: Replace the Scoring Function
« Reply #1 on: Apr 19th, 2013, 6:17am » |
Quote Modify
|
Quote:| Use a human adjudicator (or panel of adjudicators)! |
|
How would you adjudicate this game
http://arimaa.com/arimaa/gameroom/comments.cgi?gid=201392
, if gold had been able to push rabbit into a4 on 19g and just shuffle the pieces in home teritory?
Quote:| Sudden death time controls are icky for Arimaa because it is very hard to move quickly. This is a big problem IMO. |
|
As it is equally hard for both players, I don't see what's the problem with that. Moving quickly requires some skill & training 
|
| « Last Edit: Apr 19th, 2013, 10:15am by Boo » |
IP Logged |
|
|
|
Manuel
Forum Guru
Arimaa player #4020
Gender:
Posts: 58
|
|
Re: Replace the Scoring Function
« Reply #2 on: Apr 19th, 2013, 9:09am » |
Quote Modify
|
Some other cons of a human adjudicator:
- availability of a human (assuming you are not only discussing event games!)
- Also for an unbiased human it is not always easy to decide who is winning.
I actually like the idea of switching to a sudden death time-control.
|
|
IP Logged |
|
|
|
Nombril
Forum Guru
Arimaa player #4509
Gender:
Posts: 292
|
|
Re: Replace the Scoring Function
« Reply #3 on: Apr 19th, 2013, 10:33am » |
Quote Modify
|
Thanks for putting these suggestions all in one place. Since you saved your best suggestion for last, I'll save my biggest critique for last I do agree with much of what you have said, the following highlights my disagreements and additional comments.
I'm curious which two games you thought would be close to requiring resolution on score. I think it would also help if we could describe what sort of game we expect to reach the time limit. Any proposed solution should be considered in light of each of them.
In the interest of completeness, I have also seen suggestions for using the most advanced rabbits in place of the score function. Personally, I think this is a non-starter, since it is by definition gives preference to away players. An advanced rabbit can be liability or advantage depending on the position.
on Apr 19th, 2013, 3:13am, browni3141 wrote:
1. The current scoring function
Pro:
- The game will end in a known amount of time
- No site changes required
Con:
- It is going to be wrong much of the time
- Silver is arbitrarily chosen to win the tie-breaker
- Players may be encouraged to play ludicrously if they are getting close to the game ending on score
|
|
Right, giving up your elephant in order to capture two rabbits could be a winning trade if evaluated by score. BUT if we were to check the material balance for all finished games, I suspect the player with more pieces has won a vast majority of the games. From this standpoint, if we don't have a good definition of why a game to be decided on score will be so different from the general population of games, this score function is very reasonable. (NOTE: I think a game decided on score would certainly look different from the general population of games...but I don't know how to define the difference yet.)
on Apr 19th, 2013, 3:13am, browni3141 wrote:
3. The players each get to use whatever time they have remaining to make the rest of their moves
Pro:
- The game will probably not last much longer than if score were simply used
- The proper winner is much more likely to be produced.
- The game is more likely to end naturally
- This is fairly simple
Con:
- Sudden death time controls are icky for Arimaa because it is very hard to move quickly. This is a big problem IMO.
|
|
I disagree with the pro of "end naturally". If a game turns towards blitz movements, games more often end by time out or blunder. Maybe my definition of a natural end is different?
Con: This would reward a player that has banked a larger reserve, thus rewarding a time management preference rather than any playing ability. (Though I suspect any game reaching the point of needing a score resolution would see both players with max reserve remaining.)
Con: If both players start the period with same reserve, the player with faster interface and/or faster internet connection will win.
I definitely agree with your stated con. I can't see any time control related solutions being viable with current technology and online play.
on Apr 19th, 2013, 3:13am, browni3141 wrote:
And now for my (and in my opinion, the best) solution!
Use a human adjudicator (or panel of adjudicators)!
Pro:
- It will be the most accurate of all for determining the winner (accelerating time controls arguably being an exception)
- All omar has to do is say yes. No immediate site changes are required as long as the score result can be overridden.
- I volunteer to be one of the adjudicators, so lack of volunteers is no issue
- The game ends in a known amount of time
- The play of the players shouldn't be influenced near the end of the game like it would be for other alternatives
Con:
- A human might be biased (this chance can be mitigated with further methods)
Note that none of these really solves the problem of "drawn" positions. I would like to discuss that also, but that should probably be left to another thread, or at least until we talk about the main topic first.
I see no reason right now not to use a human adjudicator. My "Con" is practically meaningless considering how massively superior this is to score.
Discuss! |
|
I think the idea of a "drawn" position might be more closely linked than you want to admit. Besides positions that are close to a draw, what else would take the entire time limit? If it is a closed position where rabbits/blockades/phalanxes/etc are preventing movement, I'm not sure I'd be comfortable with an expert evaluation of who is "winning". If it is players not trying to reach victory by goal/elimination/immobilization, then I'm not too concerned about how to evaluate who won the game.
I'm also not sure that expert evaluation of board position is accurate. It is my opinion that hanzack was able to win so many games against other top players because he evaluated positional advantages differently. During many games, commentators and spectators thought hanzack's opponent had achieved a good position (frame, hostage, etc), only to later watch hanzack turn it into a devastating attack.
And my final critique:
But even if a human adjudicator can simply determine which side had a better position, then why didn't the player with the better position win the game? It actually seems ludicrous to me to award a win to a player that wasn't able to see how to use an advantage that could be seen by another player but not themselves.
|
|
IP Logged |
|
|
|
browni3141
Forum Guru
Arimaa player #7014
Gender:
Posts: 384
|
|
Re: Replace the Scoring Function
« Reply #4 on: Apr 19th, 2013, 1:48pm » |
Quote Modify
|
on Apr 19th, 2013, 9:09am, Manuel wrote:Some other cons of a human adjudicator:
- availability of a human (assuming you are not only discussing event games!)
- Also for an unbiased human it is not always easy to decide who is winning.
I actually like the idea of switching to a sudden death time-control.
|
|
I do not see the availability of a human to be a problem. I have already volunteered to do it because I know I would only be called upon once every several thousand games  I'm sure it would not be difficult to get other reasonably strong players to volunteer so that there is no problem of adjudicator availability.
I pretty much agree with Nombril on sudden death time controls. They do not treat all players equally enough to be fair, they aren't compatible with time controls with a max reserve time, and they are just no fun when it takes most players around 5-6 seconds minimum to make a move. They work better for chess because you can move in under a second and pre-moves are possible. I still prefer it over the scoring function, though.
Nombril wrote:
Quote:
I'm also not sure that expert evaluation of board position is accurate. |
|
But isn't it still more accurate than any other evaluation method? Maybe it is not perfect, but I can not think of better in terms of adjudication. I think it would be most fair to allow the adjudicator to award a draw if the position is so close that it's very hard to tell who is winning.
Quote:
If it is players not trying to reach victory by goal/elimination/immobilization, then I'm not too concerned about how to evaluate who won the game.
|
|
We should not fault a player trying to win by score because they are taking advantage of the rules. However what should we do about players who have opportunities to try to progress but do not take them, knowing that the game would be adjudicated? I disagree that this is not an important question to answer. What if it is a tournament game? The result doesn't affect just those two players. Either a draw or a double loss seems appropriate.
Quote:
It is my opinion that hanzack was able to win so many games against other top players because he evaluated positional advantages differently. During many games, commentators and spectators thought hanzack's opponent had achieved a good position (frame, hostage, etc), only to later watch hanzack turn it into a devastating attack. |
|
The fact that he won doesn't mean his early play was sound. My opinion is that a lot of the time he did get himself into a bad position and turned it into a win with skill and some luck.
Quote:
But even if a human adjudicator can simply determine which side had a better position, then why didn't the player with the better position win the game? It actually seems ludicrous to me to award a win to a player that wasn't able to see how to use an advantage that could be seen by another player but not themselves. |
|
If allowed, I would award a draw. Isn't that is the most fair result? Unfortunately, some people here seem to have a hatred towards draws, even if they would almost never occur.
If not allowed to award a draw, I guess I would give it to the player that was objectively better. Even if it's not fair it's better than rolling a die with a score function.
Since I said that I thought bias could be mitigated, and somebody brought it up anyway, then I put forth the suggestion that the two players of a game must agree on an adjudicator, and if they can not agree then one will be selected for them. This won't eliminate the chance for bias, but it will attempt to make things as fair as possible. Another suggestion to try and minimize bias is that multiple adjudicators are used and they vote.
These are the two games I was thinking of where score played a part.
http://arimaa.com/arimaa/gameroom/comments.cgi?gid=231123
I had a way to make progress but could not see it. If this were to make it to score it would be most fair for it to be a draw I think. I didn't really deserve to win. Of course I might have eventually found the way to make progress and won naturally.
http://arimaa.com/arimaa/gameroom/comments.cgi?gid=246143
Ocmiente's style seems to tend to make for very long games. The game may very well end on score and still be "alive."
What about a panel of collaborating adjudicators who are allowed to award draws and take input from the players? Majority vote would win after deliberation.
|
|
IP Logged |
|
|
|
mattj256
Forum Guru
Arimaa player #8519
Gender:
Posts: 138
|
|
Re: Replace the Scoring Function
« Reply #5 on: Apr 19th, 2013, 8:04pm » |
Quote Modify
|
Thanks for starting this browni!
My two cents:
1. Some positions are just drawn.
on Apr 19th, 2013, 12:06am, mattj256 wrote:
If the board is in that position the only fair outcome is a draw or double loss.
2. A player shouldn't be forced to sacrifice material to open up the position.
on Apr 19th, 2013, 7:48pm, mattj256 wrote:
In these two examples the position is quite drawish but both players have the option of sacrificing a rabbit to open up the board. |
|
|
|
IP Logged |
|
|
|
mattj256
Forum Guru
Arimaa player #8519
Gender:
Posts: 138
|
|
Re: Replace the Scoring Function
« Reply #6 on: Apr 19th, 2013, 9:03pm » |
Quote Modify
|
on Apr 19th, 2013, 3:13am, browni3141 wrote:I think it would be harmful to the game of Arimaa (to what degree depends on the importance of the game being played) if a game ever ends on score. Think of it as car insurance. Most of us are not expecting to need it anytime soon, but we want to be prepared for the possibility of a crash.
|
|
(1) IMO the worst-case scenario would be a bot winning the Arimaa Challenge because it won the last game on score. More generally, it would suck for any high-profile game to be decided by score. As I said in my last post, some positions are just drawn. So you're either going to have an arbitrary procedure to break ties which will sometimes be unfair, or you allow draws or double losses, or you change the rules to make it less likely that the scoring function will ever be used.
(2)
on Apr 19th, 2013, 6:17am, Boo wrote:
In the comments for the game above, Fritzlein said:
- That would have provoked an interesting discussion about how to adjudicate games that reach the time limit, specifically whether they hadn't better be called draws rather than imposing a win in an arbitrary way. I have a feeling the current rule only exists because it has never been used, and that it will only ever come into play once, because as soon as it happens once we will realize that it is the wrong rule.
Fritzlein Dec 15, 2011 2:10 pm GMT
(3) I'll make my own counter-proposals.
- If a non-rabbit piece has been immobilized (no legal moves) for X turns, that piece is removed from the board.
OR
- If a player goes X number of moves without moving a friendly rabbit forwards or pulling an enemy rabbit forwards, that player loses.
I'll let other people think about the pros and cons of these.
|
|
IP Logged |
|
|
|
Boo
Forum Guru
Arimaa player #6466
Gender:
Posts: 118
|
|
Re: Replace the Scoring Function
« Reply #7 on: Apr 20th, 2013, 3:31am » |
Quote Modify
|
Quote:| In these two examples the position is quite drawish but both players have the option of sacrificing a rabbit to open up the board. |
|
Those 2 don't look drawish.
In the first one gold can play Ee1-->f3, next move Ef3-->f4 and capture some rabbits in the next 2 moves.
In the second one gold can play Rg3-->e3+Cg1-->g3 and capture rg4 next.
Quote:| If the board is in that position the only fair outcome is a draw or double loss. |
|
If the players were to play such a game in a World Arimaa Championship in the hopes of avoiding a loss for both players, I think the outcome definetily should be the double-loss. But what if the case is not so straight-forward abuse? Where is the limit between adjudicating a draw or a double loss in an event game? Therefore I don't like the adjudication idea at all, I think scoring function is better than any adjudication, because it doesn't leave space for interpretation.
Quote:
Con: This would reward a player that has banked a larger reserve, thus rewarding a time management preference rather than any playing ability. (Though I suspect any game reaching the point of needing a score resolution would see both players with max reserve remaining.)
Con: If both players start the period with same reserve, the player with faster interface and/or faster internet connection will win.
|
|
I think this holds true only if the game reaches time limit, because both players can't find a way to make progress. (turtling-up) What if the limit is reached with both players seeing the way to progress? I suspect both players to be low on reserve then. They would have to make a choice of making a fast 'any' move, or think extra 10 sec and try to find a better one. Thus playing ability is still a big factor.
|
|
IP Logged |
|
|
|
odin73
Forum Guru
Gender:
Posts: 65
|
|
Re: Replace the Scoring Function
« Reply #8 on: Apr 20th, 2013, 2:33pm » |
Quote Modify
|
on Apr 19th, 2013, 3:13am, browni3141 wrote: And now for my (and in my opinion, the best) solution!
Use a human adjudicator (or panel of adjudicators)!
|
|
That´s surely a very intuitive and reasonable proposal concerning e.g. event games in a competition played in the gameroom.
However, since Arimaa is a strategy game with perfect information such a rule will never be an option.
I don´t know exactly what Omar´s thoughts forbidding any draw in this game, but probably a draw may serve best for the case of reaching the maximum game time. Hence, I guess that the current rule may be some trade-off to that.
Beside this: The finding of a good and reasonable function for a position taking into account number of pieces and space advantage is a very interesting issue for now and for sure in future as well.
|
|
IP Logged |
|
|
|
Fritzlein
Forum Guru
Arimaa player #706
Gender:
Posts: 5928
|
|
Re: Replace the Scoring Function
« Reply #9 on: Apr 21st, 2013, 5:54pm » |
Quote Modify
|
Yes, there is a problem. I continue to maintain, however, that we can't know the best way to fix it until we know what the problem is. Maybe we want to allow draws, because Arimaa draws occur naturally, and imposing a win on a position that is naturally drawn is too arbitrary for us to stomach. But we can't know that until we see the games.
Maybe we need sudden death. Maybe accelerating time controls are sufficient. But both of those are the wrong answer if neither player can make progress on the board, i.e. if it is a natural draw. Wouldn't it be silly for a drawn game to be determined by who could shuffle pieces faster after the position locked up? Bots would automatically get the win in any draw situation, due to faster-than-human reflexes.
Maybe there is a decent scoring function that will pick who would have won eventually. But how can we know with no examples? Or rather with only two examples, neither of which was decided on score, but both of which would have been correctly decided by the current scoring function if it had come to that. Nothing has broken yet, unless you believe that someone who doesn't know how to win a won position should not be awarded a win on score, or by arbiter, or by any other means, because a draw on the board should be a draw in fact.
To the last point, if we allow arbiters, we don't even know how to instruct them until we see the scoring function break. Do you think an arbiter who knows how to win a position the player didn't know how to win should give the clueless player the point? Is that fair, or could the other player say, hey, he couldn't beat me, so the arbiter's intervention is like giving advice? When, if ever, should an arbiter be allowed to assign a draw? When he thinks neither player would have been able to win, or only if it is drawn with perfect play? If the level of players enters into the decision, is that fair?
Again, I am not saying there is no problem. I am saying that neither you nor anyone else knows in advance how to fix the problem. We already have a rule. Probably it is a stupid rule, but at least we won't ever be stuck not knowing who has won a game, so that catastrophe is not looming. If the rule we have sucks once every ten thousand games, then we have to let the rule break at least once, to see how it breaks, before we can know what fix is appropriate. Or we can be guided by a consistent pattern of cases in which the rule almost broke. But at the moment we are essentially flying blind in trying to find a solution.
|
| « Last Edit: Apr 21st, 2013, 9:23pm by Fritzlein » |
IP Logged |
|
|
|
mattj256
Forum Guru
Arimaa player #8519
Gender:
Posts: 138
|
 |
Re: Replace the Scoring Function
« Reply #10 on: Apr 22nd, 2013, 12:03am » |
Quote Modify
|
By the way, not to change the subject, but can someone move this thread from "Site Discussion" to "General Discussion"?
Site Discussion: "Problems, suggestions or questions relating to the web site, game client, bots, etc"
General Discussion: "relating to Arimaa rules, strategy, questions, comments, announcements, etc"
|
|
IP Logged |
|
|
|
browni3141
Forum Guru
Arimaa player #7014
Gender:
Posts: 384
|
|
Re: Replace the Scoring Function
« Reply #11 on: Apr 22nd, 2013, 1:09am » |
Quote Modify
|
on Apr 22nd, 2013, 12:03am, mattj256 wrote:By the way, not to change the subject, but can someone move this thread from "Site Discussion" to "General Discussion"?
Site Discussion: "Problems, suggestions or questions relating to the web site, game client, bots, etc"
General Discussion: "relating to Arimaa rules, strategy, questions, comments, announcements, etc" |
|
By one viewpoint this discussion doesn't have to do with rules of Arimaa, but specifically how unfinished games are resolved on arimaa.com I'm not convinced it doesn't belong here. I think it could fit in either forum.
|
|
IP Logged |
|
|
|
Hippo
Forum Guru
Arimaa player #4450
Gender:
Posts: 883
|
|
Re: Replace the Scoring Function
« Reply #12 on: Apr 22nd, 2013, 6:02am » |
Quote Modify
|
on Apr 19th, 2013, 8:04pm, mattj256 wrote:Thanks for starting this browni!
My two cents:
1. Some positions are just drawn.
If the board is in that position the only fair outcome is a draw or double loss.
2. A player shouldn't be forced to sacrifice material to open up the position.
|
|
The first one is clearly lost on repetition for the player making the last rabbit step. Opponent just alternates two positions till the player remains without moves
|
| « Last Edit: Apr 22nd, 2013, 6:02am by Hippo » |
IP Logged |
|
|
|
Nombril
Forum Guru
Arimaa player #4509
Gender:
Posts: 292
|
|
Re: Replace the Scoring Function
« Reply #13 on: Apr 22nd, 2013, 10:05am » |
Quote Modify
|
on Apr 19th, 2013, 1:48pm, browni3141 wrote:| Even if it's not fair it's better than rolling a die with a score function. |
|
But it isn't rolling a die. In the vast majority of games (I'm assuming here ) the player with the most pieces wins the game. Until we can define what/how the games leading to using the score function look different, I still maintain that the score function provides a simple, predictable way to evaluate the position, that is very likely to be correct. It is always possible to find isolated counter examples, but that doesn't prove something wrong.
Oh, a potential concern for adjudication: In a bot vs human game, does this give an unfair advantage to the human player? How could a bot take into account for the evaluation function something that can't be hard defined ahead of time?
And note that all three of mattj256's examples are not draws. As Hippo pointed out, the "typical" draw of rabbits in the middle was proven by Hippo to not actually be a draw, but there will be a winner based on the repetition rule leading to "immobilization". And for the other two examples, it is easy to open the position without sacrificing material. The player just needs to be willing to take steps with pieces other than rabbits .
It sounds like the games that could be the most concerning are tournament games that should be deemed either a draw or a double-loss. So maybe that discussion is more related than I originally thought.
|
|
IP Logged |
|
|
|
browni3141
Forum Guru
Arimaa player #7014
Gender:
Posts: 384
|
|
Re: Replace the Scoring Function
« Reply #14 on: Apr 22nd, 2013, 2:13pm » |
Quote Modify
|
on Apr 22nd, 2013, 10:05am, Nombril wrote:
But it isn't rolling a die. |
|
Okay, okay, the die is loaded, but it's still a die.
The reason I like adjudication is that trying to "win by adjudication" ideally coincides with trying to win on the board. This is not the case with score. Score is not a part of Arimaa to me. It's just a hacked out safety net. Having the most pieces or the most advanced rabbit is not what Arimaa is about. It shouldn't be a factor in deciding games. The winner should be the player that actually won, or the player with better position if the game could not be completed. If we ever find a function to suit our needs here then Arimaa will be broken.
Quote:
Oh, a potential concern for adjudication: In a bot vs human game, does this give an unfair advantage to the human player? How could a bot take into account for the evaluation function something that can't be hard defined ahead of time?
|
|
I don't understand what you mean. Both players are trying to win the game. Likely ALL of the bot's evaluation is intended to help it do this. The win conditions are hard defined ahead of time. Are you trying to say that a human might try to make his position look pretty before adjudication, which is something a bot can't do? There isn't really a way around this, except to trust that the adjudicator is trying to be as objective and critical as possible.
Quote:
And note that all three of mattj256's examples are not draws. As Hippo pointed out, the "typical" draw of rabbits in the middle was proven by Hippo to not actually be a draw, but there will be a winner based on the repetition rule leading to "immobilization". |
|
Well, technically the first is not a draw, but what does you think it should be? What does your intuition tell you? From my perspective that position "is" a draw, even if it isn't
|
|
IP Logged |
|
|
|
|
Home
Help
Search
Members
Login
Register
Reply
Notify of replies
Send Topic
Print