Arimaa Forum - Print Page


    
      
        Arimaa Forum
        (http://arimaa.com/arimaa/forum/cgi/YaBB.cgi)
      

        Arimaa >> Events >> Notes about bot_Zombie in Postal Tournament
        
(Message started by: IdahoEv on Mar 1^st, 2007, 4:08pm)

Title: Notes about bot_Zombie in Postal Tournament
Post by IdahoEv on Mar 1^st, 2007, 4:08pm

seanick emailed me earlier today, concerned that zombie had not moved in a couple of days in his game. This is pretty much normal and expected, and I wanted to post in case other players were concerned.

Zombie is playing 15 games with three hours of thinking for each. That means it takes 45 hours for Zombie to cycle through and submit one turn in each game if all his opponents are submitting turns quickly. So in the early stages of the postal tournament, when people are still logging in all the time and submitting turns quickly, expect zombie to burn at least a day of reserve for every turn.

Response time to your moves may improve later in the tournament as other players go a few days without submitting moves, allowing Zombie to ignore those games for a while.

The botman script also cycles through the games in numerical order; the first one is 59204 against arimaa_master. Because I occasionally have to restart the script and it always starts from the top, games early in the list will get more attention and sometimes two moves in fairly short order.

If its operating correctly, you should only see a move every two days or so for now.

Title: Re: Notes about bot_Zombie in Postal Tournament
Post by 99of9 on Mar 2^nd, 2007, 12:50am

Are you monitoring the depth that Zombie searches to?

Even though I'm not playing against it, I'd be interested to know how deep it's going.

Title: Re: Notes about bot_Zombie in Postal Tournament
Post by Fritzlein on Mar 2^nd, 2007, 7:21am

From a glance at the logs, the short answer is that Zombie is searching about 12 or 13 steps deep.

For the opening moves, since the branching factor is lower, Zombie searched more deeply. As the branching factor rises, Zombie may not be able to search as deeply in the midgame.

After part of its thinking time is used up, Zombie decides on an initial step, i.e. it prunes all other first steps apart from its favorite, and continues searching from the second step. A while late it decides on its second step, and continues searching from the third, etc. Since Zombie might search to a depth of 12 steps after having decided on its first three steps, you could say it is searching to a depth of 15 steps, but that is a bit misleading.

I'll let IdahoEv explain the other fancy ways Zombie searches deeper/shallower in some lines. :)

Title: Re: Notes about bot_Zombie in Postal Tournament
Post by IdahoEv on Mar 2^nd, 2007, 12:23pm

This is about right; I've seen zombie search the initial position up to 17 steps deep, but only when its transposition table was much larger than it is now.

Quote:

After part of its thinking time is used up, Zombie decides on an initial step, i.e. it prunes all other first steps apart from its favorite, and continues searching from the second step. A while late it decides on its second step, and continues searching from the third, etc. Since Zombie might search to a depth of 12 steps after having decided on its first three steps, you could say it is searching to a depth of 15 steps, but that is a bit misleading.

This is essentially right, but not entirely. Zombie actually generates and searches moves, which may contain a variable number of steps. So occasionally it will select a 2-step move (a push or pull) in which case it will prune both those steps and then begin a new search from two steps later than the last one.

I actually don't think this adds any additional strength - almost inevitably the later positions zombie selects are the same as the next three steps on the branch it found for the first step. And in addition if you're going to search earlier I think you'd like to have the option of altering your move from the very first step; i.e. I think this technique assumes an independence of the four steps that is not really accurate.

I've attempted to change it so that it spent 100% of the turn searching as deep as possible for the first step, and accepted the first four steps from that tree as its move. However, everything I've attempted along those lines resulted in the occasional inexplicable suicide so clearly I'm not quite understanding unic's original search code from Fairy.

Quote:

I'll let IdahoEv explain the other fancy ways Zombie searches deeper/shallower in some lines.

Zombie has an extensions mechanic that lets it alter the target depth of a branch based on certain conditions. For example, it won't bother searching as deeply after a suicide move, on the hypothesis that a suicide usually needs to have payoff soon afterwards. After captures it tries to search a little deeper, hopefully generating some of the benefit of a quiescence search. However, the changes in depth are small (only a step or two usually) so I don't think they have a huge effect on zombie's strength.

Title: Re: Notes about bot_Zombie in Postal Tournament
Post by Fritzlein on Mar 3^rd, 2007, 9:44pm

Does anyone want to bet on how many games out of fifteen Zombie will win in the tournament? I predict Zombie will win three, for a final score of 3-12. However, I confess that about half of my expected value for Zombie's wins comes for opposing timeouts...

Title: Re: Notes about bot_Zombie in Postal Tournament
Post by arimaa_master on Mar 5^th, 2007, 3:57am

on 03/03/07 at 21:44:14, Fritzlein wrote:

I think Zombie has no chance to win against current pairing. Zombie will win only on time.

Title: Re: Notes about bot_Zombie in Postal Tournament
Post by IdahoEv on Mar 5^th, 2007, 2:56pm

I might lose against Zombie. Not on purpose, but I can still only beat Bomb 50% of the time - bots still often outtthink me tactically.

On the other hand, Zombie just gave me a camel hostage, so...

Title: Re: Notes about bot_Zombie in Postal Tournament
Post by chessandgo on Mar 5^th, 2007, 3:23pm

Zombie plays quite well, apart from this incentive to endanger its camel without purpose :( I would bet on 3 to 5 out of 15 for Zombie ...

Btw, in our two games, Zombie advanced his rabbits willingly and very early ... I knew Zombie was a lot more liberal than Bomb with it's rabbits, but here I'm a bit surprised ... maybe that's a good strategy though :)

Title: Re: Notes about bot_Zombie in Postal Tournament
Post by Fritzlein on Mar 6^th, 2007, 2:03pm

on 03/02/07 at 00:50:06, 99of9 wrote:

Are you monitoring the depth that Zombie searches to?
Even though I'm not playing against it, I'd be interested to know how deep it's going.

By the way, BombP3 (12 steps) thought for about an hour or two per move, while BombP4 (16 steps) seemed to need a day or so per move until it took a week on one move and we gave up on it.

I also looked up what Fotland said about Bomb in the postal tournament two years ago:

on 01/27/05 at 01:08:28, fotland wrote:

It doesn't search to a fixed depth. Interesting sequences are searched more deeply. In the postal games it's only looking 8 steps on some sequences, looking 14 steps typically, and looking 25 steps in the more tactical lines.

Apparently Zombie is a little slower than Bomb, if it only gets to depth 12 or 13 steps in the time it took Bomb to get 14 steps. However, it may not be that Zombie is slower, so much as that Bomb searched more selectively. Perhaps both of them examine around 300,000 nodes per second.

Title: Re: Notes about bot_Zombie in Postal Tournament
Post by Fritzlein on Apr 15^th, 2007, 7:56pm

For the first time in the tournament, Zombie is caught up on all of its games, so it has no move to think about. Of course, it eases the time pressure that Zombie has already lost three of the games, so it has only twelve remaining, but it is still set to think for three hours per move. There must have been a brief lull in the collective activity of Zombie's opposition.

In any event, there seems to be no point in having Zombie idle. Does anyone object to raising the thinking time? Technically there are supposed to be no mid-tournament modifications, and Zombie isn't smart enough to think longer for each game on its own initiative. On the other hand, given the pace at which Zombie's opponents are moving, and given the drubbing Zombie is taking in most games, I can imagine that folks would prefer a tougher, slower opponent if anything.

Title: Re: Notes about bot_Zombie in Postal Tournament
Post by RonWeasley on Apr 15^th, 2007, 8:09pm

Muggles get the same advantage as their games end. On the other hand, if players knew at the beginning that Zombie would get a little stronger as its games ended, they might have chosen to play faster. So I see good arguments either way. I suggest oppoenents who object should do so. Otherwise, let Zombie use the extra time.

IdahoEv, is there any value to you for Zombie opponents to write up a game analysis? It may be obvious where its errors are but maybe not. One common problem I see is a tendency to put the camel on the eighth rank where it is out of play.

Title: Re: Notes about bot_Zombie in Postal Tournament
Post by 99of9 on Apr 16^th, 2007, 2:57am

I'm not playing Zombie, but I'll put in my 2c anyway.

I object to changing the thinking time in the middle of the tournament. It's easier to discuss the performance of any bot if it remained identical throughout a game.

Title: Re: Notes about bot_Zombie in Postal Tournament
Post by Fritzlein on Apr 16^th, 2007, 10:36am

on 04/16/07 at 02:57:52, 99of9 wrote:

I object to changing the thinking time in the middle of the tournament. It's easier to discuss the performance of any bot if it remained identical throughout a game.

It is reasonable to want to discuss a fixed performance. I'm curious, though, why you are on the opposite side of the issue now, as compared to the issue of letting Gnobot2006CC continue to improve its opening book. Omar didn't want Gnobot2006 to keep updating its opening book with games played after the 2006 Computer Championship, on the grounds that the strength of Gnobot2006 should be a fixed reference point. If I remember correctly, you thought Gnobot2006 should be allowed to keep learning.

You wouldn't object to Zombie managing its time intelligently, with no human intervention, would you? For example, Zombie could easily determine whether it had been through the cycle quickly by looking at its remaining time for the move at hand. The decision could be as simple as this: if Zombie's clock has already run for more than a day, think three hours, otherwise think six hours.

This is important to discuss, because it gets at the intent of the "no modifications" rule. To my understanding, the intent was mostly to insure that the developer wasn't indirectly giving advice to the computer by changing the code from one game to the next. We talked about allowing mid-tournament modifications next year, as long as they weren't game-specific or move-specific modifications. Indeed, one motivation for a developer to enter a bot at all is with the hope that he will be able to improve the bot mid-tournament.

If, however, the intent of the "no modifications" rule is to make the playing strength of the bot easier to assess, that rules out catering to this particular developer motivation.

Title: Re: Notes about bot_Zombie in Postal Tournament
Post by 99of9 on Apr 16^th, 2007, 6:28pm

on 04/16/07 at 10:36:30, Fritzlein wrote:

I'm curious, though, why you are on the opposite side of the issue now, as compared to the issue of letting Gnobot2006CC continue to improve its opening book.

Changing strength in the middle of a game is a bit different to being updated in between games.

Quote:

Omar didn't want Gnobot2006 to keep updating its opening book with games played after the 2006 Computer Championship, on the grounds that the strength of Gnobot2006 should be a fixed reference point. If I remember correctly, you thought Gnobot2006 should be allowed to keep learning.

Learning was always intended to be part of the technology and capabilities of Gnobot2006. In that sense, continuing learning represents a fixed technology reference point. Since we're evaluating AI, I wanted to fix the state of the AI. I was not advocating changing the software at all. Omar has a different definition of what needs to be fixed (and was also worried about ownership issues).

Quote:

You wouldn't object to Zombie managing its time intelligently, with no human intervention, would you?

Not at all.

Quote:

We talked about allowing mid-tournament modifications next year, as long as they weren't game-specific or move-specific modifications.

I'm actually open to that if it is a policy from the start. Then we would just accept the fact that we are measuring something different.

Here it would be a weird hybrid where some games are played in full at constant strength, and other games get up to some arbitrary move and then change (perhaps more than once). What will we learn? What will the developer learn, without picking through exactly which move the switch(es) occurred on?

Title: Re: Notes about bot_Zombie in Postal Tournament
Post by Fritzlein on Apr 16^th, 2007, 10:33pm

on 04/16/07 at 18:28:21, 99of9 wrote:

What will we learn? What will the developer learn, without picking through exactly which move the switch(es) occurred on?

Oh, well, we might learn whether Zombie plays any better at six hours per move than at three hours per move. And it wouldn't be too hard to write down what all the move numbers are on the day of the change.

But honestly, the general point about allowing changes in order to give developers an opportunity to learn more was directed mostly towards next year, when the limits to the in-tournament modifications can be clearly spelled out in advance. Right now I wasn't thinking so much about what we can learn from increasing Zombie's time control. For the current tournament my thought was more about giving the human players as much of a challenge as possible with Zombie on a single computer.

Maybe others feel differently, but I personally would love to play against a bot running on a supercomputer, just to see if I could still win. I remember two years ago hoping that Fotland would increase Bomb's thinking time as much as possible, and being sorry he only raised it from two hours per move to three hours per move. I wanted to humiliate Bomb at maximum power!

Hmm... I just looked at the discussion of the 2005 Postal Tournament and found a couple of interesting quotes. ;)

on 01/27/05 at 05:47:43, 99of9 wrote:

Regarding rules of the tournament - I have no idea, but I don't think changing the time control is really a change of algorithm.

on 01/28/05 at 17:49:54, omar wrote:

I don't see any problem with allowing bomb to run for longer on each move. You still are limited by the time control of the game and using one computer to handle all the games.

But anyway, even if extending the thinking time is considered legal by precedent, I shouldn't talk so much about what I want for Zombie. I'm not even playing against Zombie, so it doesn't affect me at all, besides which it is IdahoEv's bot, not to mention IdahoEv's computer. I'll shut up now. :P

Title: Re: Notes about bot_Zombie in Postal Tournament
Post by camelback on Apr 17^th, 2007, 12:18am

Well, I'm kinda walking on a tight rope with zombie. So I would like to see if I'm fortunate enough to better the game with the current search time.

Hmm.. still a long way to go ::)
If not now, it may take a while for me to get better at zombie :)

Title: Re: Notes about bot_Zombie in Postal Tournament
Post by 99of9 on Apr 17^th, 2007, 1:59am

on 04/16/07 at 22:33:59, Fritzlein wrote:

Hmm... I just looked at the discussion of the 2005 Postal Tournament and found a couple of interesting quotes. ;)

Hehe... I'll have to practice my rhetoric before I become a politician. Scientists are allowed to change their mind every couple of years :-).

Title: Re: Notes about bot_Zombie in Postal Tournament
Post by seanick on May 14^th, 2007, 10:09am

for the record, zombie made an awesome comeback in its game vs me, to win. Sure, I made my usual larger-than-fair share of mistakes, but it played quite decently too, at the end.

I imagine the playing strength didn't actually change, but if it did, it was successful.

this was based off of Fairy, wasn't it? seems a bit stronger but then again, I lost to fairy the first time I played it too.

Title: Re: Notes about bot_Zombie in Postal Tournament
Post by Fritzlein on Jun 2^nd, 2007, 10:54am

It looks like the machine running Zombie has crashed. If IdahoEv doesn't restart it in a couple of days, I will start running Zombie from my laptop, in order to finish off Zombie's last four games. I might run it at eight hours per move to compensate for my laptop being an antique.

Title: Re: Notes about bot_Zombie in Postal Tournament
Post by IdahoEv on Jun 11^th, 2007, 1:11pm

Fortunately I got it running before I left for the weekend - the problem was a bad power strip, the circuit breaker kept blowing.

Title: Re: Notes about bot_Zombie in Postal Tournament
Post by camelback on Jun 16^th, 2007, 6:57pm

Is zombie server crashed again?
It used to make moves quickly but it's idling for more than 3 days now.

Title: Re: Notes about bot_Zombie in Postal Tournament
Post by IdahoEv on Jun 20^th, 2007, 5:47am

Yes, the server had apparently gone down ... cause unknown, the power was still flowing.

Title: Re: Notes about bot_Zombie in Postal Tournament
Post by Fritzlein on Aug 7^th, 2007, 2:18pm

What happened to Zombie's last postal tournament game, versus woh? It doesn't appear in the ongoing postal games, but it also doesn't appear in the completed games of either player.

Title: Re: Notes about bot_Zombie in Postal Tournament
Post by camelback on Aug 7^th, 2007, 4:59pm

Yeah, Strange. I saw until move 50 and it needed few more moves to finish the game.

Title: Re: Notes about bot_Zombie in Postal Tournament
Post by IdahoEv on Aug 7^th, 2007, 5:38pm

The logs look like the bot was running happily on my server, but terminated when it couldn't find an active game. Fritz, it looks like you tried restarting it just in case, am I right?

Title: Re: Notes about bot_Zombie in Postal Tournament
Post by Fritzlein on Aug 7^th, 2007, 6:54pm

No, I didn't try restarting Zombie. I figured if I couldn't see the game, Zombie probably couldn't either. I just wondered if the game somehow finished without being recorded.

Title: Re: Notes about bot_Zombie in Postal Tournament
Post by woh on Aug 8^th, 2007, 6:14am

Not sure what happened with my game against bot_Zombie. After I was trying out some moves in the plan window, the game disappeared somehow. I told Omar about the problem and he was able to recover the game.

Title: Re: Notes about bot_Zombie in Postal Tournament
Post by Fritzlein on Aug 8^th, 2007, 9:48pm

Well, anyway, it is clear that you have recovered from your earlier scare, woh. Assuming you win this game, Zombie will have a final score of 1 win and 14 losses. I confess I underestimated our human contingent, particularly in that nobody on the human side timed out. Also in the play I was impressed how both camelback and woh came back from material deficits.

Against opponents rated 1672, 1685, 1707, 1743, 1743, 1958, 1996, 1996, 2005, 2046, 2046, 2076, 2089, 2263, and 2263, a record of 1-14 is a performance rating of 1409. That is to say, a player with a rating of 1409 would be expected to win exactly one game against this competition, according to the rating formula.

This is a rather pale showing for Zombie compared to Bomb's performance rating of 1716 in the 2005 Postal tournament, with a record of 4 wins and 6 losses. I don't think the difference can be chalked up to rating deflation, i.e. the notion that a 1700 player back then would be rated less than 1700 today. On the contrary, I expect that if Bomb2005 enters the next postal tournament, it will have a performance rating around 1700 all over again.

Nor do I think the difference can be due to the hardware running Bomb versus the hardware running Zombie, especially since Zombie thought for twice as long as Bomb.

On the contrary, it seems that Bomb simply has a decent grasp of strategy, particularly of trap control. Whether this is due to the quiescence search or other factors, I don't know. Also, Zombie's willingness to rush with rabbits and camel may be scarier in a live game than in a postal game; Bomb's defensive style is more suited to slow games.

I'm afraid that this shows not only that Zombie has a long way to go to challenge humans, but also that Zombie has a long way to go to challenge Bomb. :( Furthermore, as long as Bomb remains strongest without improvements, Fotland will probably not devote time to making Bomb better. :( :(

Title: Re: Notes about bot_Zombie in Postal Tournament
Post by woh on Aug 15^th, 2007, 6:59am

on 08/08/07 at 21:48:03, Fritzlein wrote:

Well, anyway, it is clear that you have recovered from your earlier scare, woh.

Things sure weren't looking bright for me at some point in the game. I was lucky to come out on top.

It looks like Zombie shouldn't hang on to a framed rabbit as long as he is. In both my PT games against Zombie I was able to capture 3 or 4 rabbits while Zombie was unwilling to give up the framed rabbit.

Title: Re: Notes about bot_Zombie in Postal Tournament
Post by Fritzlein on Aug 15^th, 2007, 7:30am

It was a treat to have a bot playing in the postal tournament again. If nothing else it shows that bots really do beat us because of short-term tactics that we overlook, and when it comes to strategy, bots have very little clue. Arimaa makes me feel good about my mental faculties compared to silicon, even though I expect it is just a matter of time before the machines pass me up.

Title: Re: Notes about bot_Zombie in Postal Tournament
Post by IdahoEv on Aug 15^th, 2007, 4:48pm

And it was likewise fun sponsoring a bot in the tournament. Even though I've made only a very few changes from Fairy (the bot of which Zombie's code is a fork) so far, it was interesting to feel like I had a spiritual protege competing.

I've had too busy of a summer to code much, but hopefully soon I'll have time to finish porting Zombie into Java so that I can use my research code to take it to thee next level for the 2007/2008 tournament season.

Title: Re: Notes about bot_Zombie in Postal Tournament
Post by Fritzlein on Aug 23^rd, 2007, 1:50pm

Say, IdahoEv, it looks like Zombie is still logging into the game room all the time. Does it not know the tournament is over?

Title: Re: Notes about bot_Zombie in Postal Tournament
Post by IdahoEv on Aug 24^th, 2007, 2:05pm

Weird, I thought it was off.

I just shut it down. If it shows up again, it will have earned the name for sure!