Author |
Topic: Arimaa rating deflation (Read 30021 times) |
|
omar
Forum Guru
Arimaa player #2
Gender: 
Posts: 1003
|
 |
Re: Arimaa rating deflation
« Reply #120 on: Jan 25th, 2008, 5:54pm » |
 Quote  Modify
|
I didn't quite get your scoring formula at first, but Karl pointed out that it basically amounts to only maximizing the players score and using the opponents score only to break ties.
In that case there is a difference and we can't assume that the rating of ArimaaScoreP1 is 1074. Since you are computing both players score anyways, would it be easy enough to modify the program to include a bot that uses the same eval as ArimaaScoreP1. I haven't been able to get the program to compile on linux.
Here is the link to the original program:
http://arimaa.com/arimaa/download/randomBot/claude/r.cpp
If someone could get this to compile with gcc that would be great.
|
| « Last Edit: Jan 25th, 2008, 6:07pm by omar » |
 IP Logged |
|
|
|
clauchau
Forum Guru
bot Quantum Leapfrog's father
Gender:
Posts: 145
|
|
Re: Arimaa rating deflation
« Reply #121 on: Jan 28th, 2008, 4:29am » |
Quote Modify
|
on Jan 25th, 2008, 5:54pm, omar wrote:
Thanks for the link, I had lost it all. I was able to get it compiled with gcc by adding :
Code:
and explicitly casting back negated enums = ints to enums :
Code:| const Direction Right= (Direction)-Left; |
|
I've coded ArimaaScoreP1 and ran 1000 games against M+S-S. It only takes 4 minutes here, so I'll run more later when I've fixed something about too long games.
The way it is now, ArimaaScoreP1 won 250 games, M+S-S won 723 games, and 27 games were counted as draws because they lasted more than 512 plies. I have to fix these draws and use the Arimaa score to count more winning games, don't I?
Also I wonder what ArimaaScoreP1 does about repetitions. Over my 1000 games, 49 ended because of a position repeated a 3rd time, because none of the bots in my experiment care about repetition. Should I leave my ArimaaScoreP1 like that, not caring ?
|
|
IP Logged |
|
|
|
clauchau
Forum Guru
bot Quantum Leapfrog's father
Gender:
Posts: 145
|
|
Re: Arimaa rating deflation
« Reply #123 on: Jan 28th, 2008, 10:19am » |
Quote Modify
|
Thanks Omar.
Good, I'll check the source and prevents ArimaaScore to repeat according to what I find. It seems bot_ArimaaScore avoids first-time repeatitions indeed but I'll check more thoroughly.
I'll also get its initial setup move right tomorrow, I have it wrong so far.
Before getting it right, I ran 10,000 random games : 316 games reached my 512 plies limit (256 Gold moves and 256 Silver moves). They really were draws and the ArimaaScore didn't matter for any of them because they all consisted of roaming lone Gold and Silver elephants.
|
|
IP Logged |
|
|
|
omar
Forum Guru
Arimaa player #2
Gender:
Posts: 1003
|
|
Re: Arimaa rating deflation
« Reply #124 on: Jan 29th, 2008, 8:05am » |
Quote Modify
|
on Jan 28th, 2008, 10:19am, clauchau wrote:| Before getting it right, I ran 10,000 random games : 316 games reached my 512 plies limit (256 Gold moves and 256 Silver moves). They really were draws and the ArimaaScore didn't matter for any of them because they all consisted of roaming lone Gold and Silver elephants. |
|
Since we have been playing all our tournament games using the extermination rule already and I plan to make that the default for rated games in the gameroom, lets apply that for these games as well. The first player to lose all the rabbits loses the game by extermination. It will eliminate the draws that you are seeing. Draws could still be possible if both players have lost their last rabbit after the completion of the move, but this would be super super rare.
|
|
IP Logged |
|
|
|
jdb
Forum Guru
Arimaa player #214
Gender:
Posts: 682
|
|
Re: Arimaa rating deflation
« Reply #125 on: Jan 29th, 2008, 8:59am » |
Quote Modify
|
on Jan 29th, 2008, 8:05am, omar wrote:
Draws could still be possible if both players have lost their last rabbit after the completion of the move, but this would be super super rare.
|
|
I thought that was a win for the player making the move, or am I mistaken?
|
|
IP Logged |
|
|
|
clauchau
Forum Guru
bot Quantum Leapfrog's father
Gender:
Posts: 145
|
|
Re: Arimaa rating deflation
« Reply #127 on: Jan 30th, 2008, 9:00am » |
Quote Modify
|
As Don Dailey admitted, bot_ArimaaScore and other parent bots pick up a move among equivalent best moves in a biased random way. You can see the bias demonstrated when shuffling a few cards on "Coding Horror: The Danger of Naivete": http://www.codinghorror.com/blog/archives/001015.html
The more cards, the bigger the bias. It may result in picking up some moves a magnitude more often. The bias is difficult to foresee because it depends on how many equivalent moves there are and the anecdotical order in which the bot initially builds up the move list.
I'm totally happy with it as far as playing goes, and I don't really know whether it makes the bot any pourcent stronger or not, but I'm not comfortable with it when building up reproducible basic knowledge about Arimaa and simple reference bots. So I plan to gather statistics about an idealized bot_ArimaaScoreP1 where that bias is corrected. I'll test whether my own version of a biased bot_ArimmaScoreP1 is any stronger than the unbiased one.
In case it matters, the correction is easy to make in the source code. We have to change Line 927 or so in the function scramble of getMove :
Code:| int r = rand() % lc; // swaps the current card with any card in the deck |
|
should become
Code:| int r = i + rand() % (lc-i); // swaps the current card with itself or any following card |
|
|
|
IP Logged |
|
|
|
clauchau
Forum Guru
bot Quantum Leapfrog's father
Gender:
Posts: 145
|
|
Re: Arimaa rating deflation
« Reply #128 on: Jan 30th, 2008, 10:19am » |
Quote Modify
|
on Jan 29th, 2008, 8:05am, omar wrote:| The first player to lose all the rabbits loses the game by extermination. |
|
Cool, this makes a big difference with basic bots and gives more victories to the bots that deserve them more. In my random sampled matches, I don't get any more game drawn by being too long or any game lost by repetition, because the rabbits get extinct before.
|
|
IP Logged |
|
|
|
mistre
Forum Guru
Gender:
Posts: 553
|
|
Re: Arimaa rating deflation
« Reply #129 on: Jan 30th, 2008, 10:33am » |
Quote Modify
|
Omar,
With the new extermination rule, this makes it impossible to have a draw, correct?
Once implemented, you can change the reason code n = no rabbits left; draw to e = extermination, player has no rabbits left.
|
|
IP Logged |
|
|
|
clauchau
Forum Guru
bot Quantum Leapfrog's father
Gender:
Posts: 145
|
|
Re: Arimaa rating deflation
« Reply #130 on: Jan 30th, 2008, 10:44am » |
Quote Modify
|
on Jan 28th, 2008, 9:07am, omar wrote:http://arimaa.com/arimaa/bots/bot_ArimaaScore/src/setups
|
|
Oh well, now that I've argued for a simple idealized bot_ArimaaScoreP1, especially because I would have a hard time to reproduce the current bias in scrambling its move list, I also feel like questioning the initial setup the real bot uses on the site. It has nothing to do with maximizing the Arimaa Score. It's using some other human knowledge and common setups of other bots. I'd rather keep having it to maximize the Arimaa Score and put all the rabbits on the second row.
However I'll do my best in fighting my reluctance to study the real bots if I feel it is wanted. What do you think, is it better to anchor the rankings on a real actual bot, with tons of games already played, or on a new simple idealized version that could be made available as well but with an empty playing record in the gameroom, with the advantage that it is more easily described and reproductible?
|
| « Last Edit: Jan 30th, 2008, 10:45am by clauchau » |
IP Logged |
|
|
|
Fritzlein
Forum Guru
Arimaa player #706
Gender:
Posts: 5928
|
|
Re: Arimaa rating deflation
« Reply #131 on: Jan 30th, 2008, 10:47am » |
Quote Modify
|
Even with the rabbit extermination rule, there is a possibility the players will shuffle pieces endlessly without accomplishing anything. Technically the repetition rule will end such games eventually, so they aren't drawn by rule, but in practice the sun might burn out before the game ends. The practical solution for live games, at least at present, is to have a time cutoff after which the ArimaaScore formula determines the winner. But what if the scores of the two players are tied? Omar has foreseen even this infinitesimal possibility, and ruled that Silver wins if the score is tied after time cutoff.
At one time we feared that optimal play would produce indefinite piece shuffling, but now it seems that optimal play will either pull opposing rabbits or advance friendly rabbits voluntarily, so there is little chance of drawn out human games. Bots are another matter. They are still dumb enough to potentially get caught playing aimlessly forever. Thus Claude needs to have some move cutoff built into his testing program.
I think what Claude's latest post is saying is that the bots he is testing now are so dumb and aggressive (not smart and defensive) that some bot will lose all its rabbits before the cutoff comes into play.
|
|
IP Logged |
|
|
|
Fritzlein
Forum Guru
Arimaa player #706
Gender:
Posts: 5928
|
|
Re: Arimaa rating deflation
« Reply #132 on: Jan 30th, 2008, 11:06am » |
Quote Modify
|
on Jan 30th, 2008, 10:44am, clauchau wrote:| What do you think, is it better to anchor the rankings on a real actual bot, with tons of games already played, or on a new simple idealized version that could be made available as well but with an empty playing record in the gameroom, with the advantage that it is more easily described and reproductible? |
|
I like the idea of anchoring on an idealized bot that is easier to describe. ArimaaScoreP1 plays stronger if you give it a fixed setup with all rabbits back, but so what? It would also play stronger with a fixed first move of elephant forward four steps. The objective isn't to have a strong bot, the objective is to give beginners a punching bag so they can familiarize with the game.
The idealized bot is one that chooses the move that maximizes ArimaaScore, and if there is more than one such move, chooses each with equal probability. For setup this means all rabbits forward and the other pieces randomly shuffled on the back row. The only reason to make an exception for the setup would be to make the bot play stronger.
However, if we have a choice of two idealized anchors, both equally easy to describe, I would prefer to use the stronger anchor. Claude, did you say that the bot which played to maximize its own ArimaaScore was stronger than the one which played to maximize its own ArimaaScore minus the opposing ArimaaScore?
I'm not even wedded to the idea that the anchor must be ArimaaScoreP1. There may be a stronger one-ply bot with a simpler description, in which case I would vote to add that bot to the ladder as the anchor instead, rather than anchoring on an existing bot. But in the strength vs. simplicity tradeoff, I would definitely not want to use anything searching deeper than one ply.
|
|
IP Logged |
|
|
|
Fritzlein
Forum Guru
Arimaa player #706
Gender:
Posts: 5928
|
|
Re: Arimaa rating deflation
« Reply #133 on: Jan 30th, 2008, 11:29am » |
Quote Modify
|
Incidentally, if you wonder why I formerly opposed using anchor bots but now support it, my stance actually has little to do with anchoring per se. I used to believe that anchor bots would actually contribute to ratings inflation, because people would find an anchor they could beat and beat it a zillion times. Since they gain but the anchor doesn't lose, that's a net injection of points into the system.
However, other folks have pointed out that we seem to be suffering from inflation in the current system anyway, and I am starting to believe it. What is the root cause of this inflation? I think Ryan Cable was the first to point out it is coming from the bottom of the bot ladder. New users join, play a few games, and quit with a lower rating than they started. That means they left behind some points in the system.
Now, if the lowest bot in the ladder is anchored rather then floating, a new user who loses to that bot will not leave any points behind. That bot's rating will not budge. So the single biggest injection of points has been eliminated at a stroke.
So you see I still don't care about anchoring the system per se. I just want an anti-inflationary measure. If we anchored the rating of BombP2, I would hate it, because I would expect it to be inflationary, making our problems worse rather than better. Anchoring the lowest bot is good only because of the way our ladder works to make that anti-inflationary.
If anchoring the lowest bot happens to have a fringe benefit that we can say our rating scale is somehow related to a random mover having a rating of zero, that's an added bonus, but it isn't my reason for supporting the change. 
|
|
IP Logged |
|
|
|
clauchau
Forum Guru
bot Quantum Leapfrog's father
Gender:
Posts: 145
|
|
Re: Arimaa rating deflation
« Reply #134 on: Jan 31st, 2008, 7:31am » |
Quote Modify
|
on Jan 30th, 2008, 11:06am, Fritzlein wrote:| Claude, did you say that the bot which played to maximize its own ArimaaScore was stronger than the one which played to maximize its own ArimaaScore minus the opposing ArimaaScore? |
|
Almost. More exactly, with every elementary one-sided score I've tried, including ArimaaScore, the bot which played to maximize its own score first and in case of equality minimized the opposing score was stronger than the one which played to maximize its own score minus the opposing score. With (idealized) ArimaaScore, the 1st bot wins about 65% of its games against the 2nd bot.
It seems to show that attacking more than defending is rewarding in Arimaa.
|
|
IP Logged |
|
|
|
|
Home
Help
Search
Members
Login
Register
Reply
Notify of replies
Send Topic
Print