Arimaa Forum
        (http://arimaa.com/arimaa/forum/cgi/YaBB.cgi)
      

        Arimaa >> Events >> 2009 Computer Championship
        
(Message started by: Fritzlein on Jan 22nd, 2009, 3:47pm)
Title: 2009 Computer Championship
Post by Fritzlein on Jan 22nd, 2009, 3:47pm
I notice that of the registered participants for the 2009 Computer Championship, bot_bomb is second-to-bottom with a rating of 1584.  This rating will be three years old by the time the tournament is seeded.  Assuming that the bot entered will be unchanged from bot_Bomb2005CC, I propose that the rating used to seed Bomb in the tournament should instead be bot_Bomb2005CC's rating.  More accurate seeding will make the tournament more fair.

Of course, if it won't be exactly the same code, then it would be unfair to use the rating of bot_Bomb2005CC, and we would have no better choice than to use the low rating of bot_bomb.  But for an unchanged bot it seems unfair that it wouldn't even get the benefit of its victories in the 2007 and 2008 Computer Championship and Arimaa Challenge cycles.
Title: Re: 2009 Computer Championship
Post by 99of9 on Jan 22nd, 2009, 4:03pm
I agree that this makes sense.
Title: Re: 2009 Computer Championship
Post by Fritzlein on Jan 22nd, 2009, 4:30pm

on 01/22/09 at 16:03:34, 99of9 wrote:
I agree that this makes sense.

Actually, this was your idea, wasn't it?  So I should be saying that I agree with you.  :)
Title: Re: 2009 Computer Championship
Post by 99of9 on Jan 22nd, 2009, 5:16pm
In the words of Fritz: "It seems that if the bot is unaltered from 2005, it should at least get the benefit of Bomb2005's rating for seeding."

In the words of the follower: "yes, i agree that if omar enters bomb2005cc in the tournament, the real rating should come with it"
Title: Re: 2009 Computer Championship
Post by Fritzlein on Jan 22nd, 2009, 8:16pm
My memory is so poor... :-[
Title: Re: 2009 Computer Championship
Post by 99of9 on Jan 22nd, 2009, 9:00pm
[quote author=Fritzlein link=board=talk;num=1232668024;start=0#4 date=01/22/09 at 20:16:47]My memory is so poor... :-[/quote]
Hopefully by the time we next play in the WC you will have totally forgotten how to play arimaa.  :o
Title: Re: 2009 Computer Championship
Post by Fritzlein on Jan 31st, 2009, 5:16pm
Omar mentioned in the chat that if only seven bots have registered by the deadline, he intends to round out the field to eight with Occam or some other bot.  This is a nice gesture on his part, and I fully support it.  However, given Occam's weak showing in 2007, I would recommend using a stronger bot if publicly available, by which I mean Fairy or Zombie.  The level of bots these days has improved so much that playing Occam is too close to a bye, and if we're just handing out byes we would just as well have seven computers.

That's my intuition, anyway.  How do other people feel about having a relatively strong bot entered automatically as a benchmark?  Is that unfair to the registered competitors?
Title: Re: 2009 Computer Championship
Post by arimaa_master on Feb 1st, 2009, 7:13am

on 01/31/09 at 17:16:36, Fritzlein wrote:
How do other people feel about having a relatively strong bot entered automatically as a benchmark?  Is that unfair to the registered competitors?


I fully support this idea, although the fairness question should be answered by bot developers before that.
Title: Re: 2009 Computer Championship
Post by Janzert on Feb 1st, 2009, 11:09am
I guess I don't particularly mind any of the cases (strong bot, weak bot or no 8th bot) and can see arguments for and against each. So whatever is decided is fine with me.

Janzert
Title: Re: 2009 Computer Championship
Post by jdb on Feb 1st, 2009, 4:16pm
I hope we can get an eighth bot. There will be too many byes with 7 bots.
Title: Re: 2009 Computer Championship
Post by 99of9 on Feb 1st, 2009, 4:34pm
I would also be happy to see an eighth bot, but would prefer to see a currently developed one if possible.
Title: Re: 2009 Computer Championship
Post by omar on Feb 2nd, 2009, 8:24am
I've extended the bot registration deadline in case any developers missed it, but wanted to register. The max I would be able to extend it is until Feb 6th, since I need at least a day or two to get the accounts setup. I really would like to see an actively developed bot as the 8th entry rather than one of the bots from previous years.
Title: Re: 2009 Computer Championship
Post by aaaa on Feb 2nd, 2009, 8:33am
Sorry if I have come across as a bit passive-aggressive in this regard, but in case it hasn't already been made clear now, I'm not entering my bot in the championship as I don't consider it up to par for it yet.
Title: Re: 2009 Computer Championship
Post by omar on Feb 3rd, 2009, 8:54am
You really should reconsider aaaa. Your bot is doing pretty well against Badger, and has previously won against many of the other bots. I'll sponsor your bot in the same way that Fritzlein sponsored Badger :-)
Title: Re: 2009 Computer Championship
Post by Fritzlein on Feb 4th, 2009, 1:16pm
For this year a prediction contest on the Computer Championship would be more interesting than for the World Championship.  What have we had in the World Championship: over 40 games but only three wins by the lower-rated player?  The Computer Championship will be a mere 22 games or so, but I'll bet the lower-rated player will win more than three of those.

I'm surprised nobody is submitting odds like me, because this is a perfect chance to get it completely wrong and look like a fool.  Who can pass on that?  :D
Title: Re: 2009 Computer Championship
Post by omar on Feb 24th, 2009, 12:44pm
When I went to order the second server for the computer championship, I found that the hosting company no longer offers a system with the same hardware configuration as the first server. So I had to order two new servers with the same hardware configuration. I will be transferring the bot accounts over to the new servers today. I had to disable logins to the accounts while I do the transfer. I will email you new login info when when transfers are complete.
Title: Re: 2009 Computer Championship
Post by omar on Feb 24th, 2009, 3:16pm
New login info has been sent. If you did not get it, send me a message through the Contact page.
Title: Re: 2009 Computer Championship
Post by 99of9 on Feb 24th, 2009, 3:30pm

on 02/24/09 at 12:44:25, omar wrote:
So I had to order two new servers with the same hardware configuration.

Intel(R) Core(TM)2 Quad  CPU   Q8200  @ 2.33GHz

It's a quad core!  Bye bye bomb?  Thanks for the upgrade omar.  Although it must be a bit disappointing to Janzert who was working on the assumption that it would be a Duo.
Title: Re: 2009 Computer Championship
Post by omar on Feb 24th, 2009, 4:05pm
I didn't actually upgrade. The hosting company discontinued the previous hardware and offered this in its place.
Title: Re: 2009 Computer Championship
Post by Fritzlein on Feb 24th, 2009, 5:10pm
I have to revise my odds.  I'm guessing that the quad core will give a 120-point rating advantage to the parallel bots over the non-parallel bots.
Title: Re: 2009 Computer Championship
Post by Janzert on Feb 24th, 2009, 5:45pm
Ouch, yes not just a quad core but the individual core speed went down from 3Ghz to 2.33. Ah well I knew that parallelizing it was something I needed to do right after the CC anyway.

Janzert
Title: Re: 2009 Computer Championship
Post by 99of9 on Feb 24th, 2009, 9:32pm
Omar, there are conflicting dates on when we need to have submitted our final programs by.  In one place it says the 25th, but in another it says the 28th (http://arimaa.com/arimaa/wcc/2009/rules.html).

Could I request that you give us until the 28th in order to allow a little time to adapt to the new server?

Thanks
Toby
Title: Re: 2009 Computer Championship
Post by Janzert on Feb 24th, 2009, 10:15pm
Where is the 25th mentioned? I've certainly been planning on the 28th listed near the top of the rules page.

Janzert
Title: Re: 2009 Computer Championship
Post by 99of9 on Feb 25th, 2009, 2:25am

on 02/24/09 at 22:15:22, Janzert wrote:
Where is the 25th mentioned?

http://arimaa.com/arimaa/wc/2009/sch.html
Title: Re: 2009 Computer Championship
Post by omar on Feb 25th, 2009, 8:39am
Thanks for letting me know about this Toby. I changed both pages to be the same and increased the time a bit. So instead of before Feb 28th, it is now before 6PM EST Feb 28th.
Title: Re: 2009 Computer Championship
Post by Fritzlein on Feb 26th, 2009, 6:40am
Whoops, I tried to reply to my own post #14, and accidentally over-wrote it instead.  I'm not trying to destroy the record of my prediction, which as near as I can remember was

35% Bomb
30% Clueless
20% Gnobot
10% Opfor
5% Sharp, Badger, Zombie, Rat

Given the quad core I'm going to bump up Clueless and GnoBot for a new prediction:

40% Clueless
25% Bomb
25% Gnobot
5% Opfor
5% Sharp, Badger, Zombie, Rat

Someone else should jump in with their predictions, or else I automatically am the best predictor, and I win the Spectator Contest by default.  :)
Title: Re: 2009 Computer Championship
Post by jdb on Feb 26th, 2009, 7:34am
Using the current ratings of the bots and the tournament simulator with rating uncertainty of 100:

 1   clueless 60.3%
 2   gnobot 30.9%
 3    opfor 4.3%
 4    bomb 3.9%
 5    badger 0.3%
 6    zombie 0.1%
 7    sharp 0.1%
 8    rat 0.1%
Title: Re: 2009 Computer Championship
Post by RonWeasley on Feb 26th, 2009, 9:15am
Bomb has a lot of experience in the big matches.  The rookie bots aren't used to the venue and are prone to mistakes.  So I'm going with:

Bomb  40%
Clueless  35%
Gnobot  34%
Opfor  31%
Badger  30%
Zombie, Sharp, Rat  25%

The fact that these don't add up to 100% is due to round-off error.
Title: Re: 2009 Computer Championship
Post by Janzert on Feb 26th, 2009, 10:23am
Makes a note to only to ever get in a flight simulator designed by RonWeasley.

Janzert
Title: Re: 2009 Computer Championship
Post by Tuks on Feb 28th, 2009, 2:13am
ide say its about

Clueless 40%
Gnobot 25%
Bomb 15%
Opfor 15%
Badger 4%
Rest 1%

well...badger beat gnobot so im inclined to increase its chance and reduce gnobots chance because i havent seen anyone else lose to badger yet
Title: Re: 2009 Computer Championship
Post by 99of9 on Feb 28th, 2009, 4:34am
Gnobot 99%
Rat 1%

;)
Title: Re: 2009 Computer Championship
Post by BlackKnight on Feb 28th, 2009, 1:23pm
Agreed.  ;)
Title: Re: 2009 Computer Championship
Post by aaaa on Mar 1st, 2009, 10:26am
Shouldn't the time between games be at least as large as their maximum possible duration, in this case 8 hours? Otherwise, bots would start sharing CPU cycles if the games overlap, right?
Title: Re: 2009 Computer Championship
Post by Tuks on Mar 1st, 2009, 10:29am
especially since there are bots like rat who like to prolong games unnecessarily :)
Title: Re: 2009 Computer Championship
Post by 99of9 on Mar 1st, 2009, 11:43am
Also, Gnobot's post-game script will hog computer resources and may cause another bot running to time out.
Title: Re: 2009 Computer Championship
Post by omar on Mar 1st, 2009, 5:23pm
Last year the longest game was 56 moves and took about 3.5 hours to finish. Most of the other ones finished within 3hrs.

I usually set them up to start automatically, but this year I didn't set it up that way and have to issue a command to start the game. So I'll be keeping an eye on it and will push the other games out if one is going longer then usual.

Spectators can keep an eye on the server load and processes running from these pages:

http://gold.arimaa.com/logs/

http://silver.arimaa.com/logs/
Title: Re: 2009 Computer Championship
Post by omar on Mar 1st, 2009, 9:08pm
Tournament
--
I was able to play some games against the CC bots when they were online. Based on how hard a time the bots gave me I would have to say that clueless would be my first choice for winning the tournament this year followed closely by GnoBot. Of course the developers have been working on the bots till the very end and I never played any of the bots at the full 2 min per move on the quad core hardware so their performance could be way different then what I am guessing. But one thing for certain is that this years tournament is very different than the ones we've seen in the past. Unlike previous years where bot_Bomb had very high chances of winning I think it's chances of winning the tournament this year are pretty low. It is quite amazing the David Fotland developed within one year a bot that could stay ahead of the pack for five years. This year is also going to mark a turning point for the Arimaa bots. To be competitive the bots will have to start making use of multi-core CPUs. Since getting more cycles per second is getting increasingly harder, this is going to be the only way for bots to ride the hardware improvement ramp. Humans have enjoyed a six year plateau in bot performance and perhaps gotten a little over confidence in our ability to stay ahead of the bots. Now the fun is really going to begin and we are going to be tested like never before. My expectation is that humans are going to learn a lot about tactics from the bots. Also we are going to learn something intrinsic about the Arimaa game tree. First we are going to notice that there are areas of the game tree that are pro tactics and areas that are pro strategy. Then the real battle for humans is going to be to see if they can steer the game into a pro strategy area while the bots try to steer it into a pro tactics area. If the Arimaa game tree is intrinsically more pro strategy then the humans will be able to maintain the lead; otherwise the bots will close the gap and potentially surpass us just by riding the hardware improvement ramp.


Bot Sharp
--
Although David Wu has been making a lot of improvement to bot_Sharp, he felt that it was not quite ready for this years tournament and decided to use last years version. Here is what he said to me in a recent email:

Quote:
... I have a current version of the program that is well-tested and far more robust and bug-free than the original in every part that is implemented (almost all of it, in fact), as well as more flexible (actually paying attention to time controls, and such), but I don't quite think I will be able to finish in the next week. As such, I would appreciate it if you could set up last year's version. ...


Wiki
--
I totally forgot to ask if someone could help with the event coverage for the computer championship. I was pleasantly surprised to see that the wiki page for the computer championship was already being updated.  If you can let me know who the main contributors are I'll add their names to the tournament page as event reporters. Thanks for the reporting guys :-)
Title: Re: 2009 Computer Championship
Post by Janzert on Mar 1st, 2009, 9:43pm
99of9 did the work of putting together the CC event page on the wiki and has done a great job. Adanac has said that he will be able to fill in a game or two for each round. I will try and fill any other game reports that need it. I hope others will feel free add to or modify any that I do though. Remember it's a wiki, please jump in anywhere you want.

Janzert
Title: Re: 2009 Computer Championship
Post by Fritzlein on Mar 2nd, 2009, 9:28am

on 03/01/09 at 17:23:36, omar wrote:
Spectators can keep an eye on the server load and processes running from these pages:

http://gold.arimaa.com/logs/

http://silver.arimaa.com/logs/


The logs on silver.arimaa.com clearly show that Rat never stopped running.  It was eating up one CPU even after the next game started, so clueless only had three CPUs to work with instead of four against sharp.  The game should be aborted if it is still going on, and invalidated if we catch it too late.
Title: Re: 2009 Computer Championship
Post by omar on Mar 2nd, 2009, 10:05am
I've sent the following message to the TD and will wait on his response to see what we should do.


Quote:
Prior to the bot_Sharp2008CC vs bot_clueless game the
bot_GnoBot vs bot_Rat game occured and finished with
bot_Rat losing due to time. I checked the arimaa.com
server and did not find any problem that could have
caused the time out. However, I did not notice that
bot_Rat was still continuing to run and did not exit
after the game timed out.

After the bot_Sharp2008CC vs bot_clueless had started
Karl noticed that bot_Rat was still running and
notified me. I immediately killed the bot_Rat process
so that it does not consume any resources and effect
bot_clueless. It was around move 13s when I killed
the bot_Rat process. I then tried to call you, but
was informed that you were out.

I will let the game finish, but just wanted to notify
you about this situation. I'll also post a message
on the forum.
Title: Re: 2009 Computer Championship
Post by Janzert on Mar 2nd, 2009, 10:32am
I would think since it could only hurt clueless' play and clueless went on to win anyway that it didn't have an effect on the game outcome.

Janzert
Title: Re: 2009 Computer Championship
Post by omar on Mar 2nd, 2009, 11:41am
True; and even at the time I became aware of the problem clueless was in a strong position.

But after thinking about this, I think I see why Fritzlein suggested stopping the game. If the game had turned out that clueless lost; the TD most likely would have decided that it should be replayed. Thus, clueless would be getting two chances to win the game. So it makes sense to stop the game immediately if we become aware of such a problem.
Title: Re: 2009 Computer Championship
Post by aaaa on Mar 2nd, 2009, 12:42pm
I think there should be an automatic process that checks whether the computers are completely unoccupied before it starts a championship game.
Title: Re: 2009 Computer Championship
Post by Fritzlein on Mar 2nd, 2009, 2:15pm

on 03/02/09 at 11:41:59, omar wrote:
But after thinking about this, I think I see why Fritzlein suggested stopping the game. If the game had turned out that clueless lost; the TD most likely would have decided that it should be replayed. Thus, clueless would be getting two chances to win the game. So it makes sense to stop the game immediately if we become aware of such a problem.

Yes, I think the principle of not getting two chances is essential.  One could argue that it is not fair to clueless to restart after 13 moves because clueless already had a great position.  But if clueless had had a bad position, then we surely would have restarted the game.  This means that losing a CPU was a bonus for clueless rather than a detriment.  Either it gets to play on from a winning position, or it gets a free restart.  That's not fair to sharp.

I think that even now the game should be invalidated and replayed for the same reason.  I think everyone agrees that we would have invalidated the result if clueless had lost, so the only fair thing to do is invalidate the game even though clueless won.

And for future reference, next time a problem is discovered mid-game, I highly recommend that the game be suspended immediately.  We always have the option of resuming from the suspended position if that later on seems like the fairest thing to do.  The computers won't use the extra time to rest or strategize or gain any kind of advantage in between.  The bad thing about playing on before making a decision is that delay can create the perception (if not the reality) that the ultimate decision was affected by how the rest of the game played out.  That applies not only to this specific case, but other situations as well.
Title: Re: 2009 Computer Championship
Post by Fritzlein on Mar 2nd, 2009, 2:18pm

on 03/02/09 at 10:05:10, omar wrote:
Karl noticed that bot_Rat was still running

Credit where credit is due; Janzert is the one who noticed Rat was still running; I just read his comment in chat.
Title: Re: 2009 Computer Championship
Post by 99of9 on Mar 2nd, 2009, 4:15pm

on 03/02/09 at 14:15:50, Fritzlein wrote:
And for future reference, next time a problem is discovered mid-game, I highly recommend that the game be suspended immediately.  We always have the option of resuming from the suspended position if that later on seems like the fairest thing to do.

For the sake of gnobot's book, I would not like to see a winner declared in the middle of an otherwise functional game (or does omar have the facility to "suspend" in the true sense of the word?)

Is it possible to make sure such games do not get into the games database (at least for the duration of the tournament)?
Title: Re: 2009 Computer Championship
Post by RonWeasley on Mar 2nd, 2009, 4:27pm
I'm ruling that clueless vs sharp be replayed.  I think this is definitely in "discretion" territory, but not replaying raises harder questions than not in my opinion.

Technically, I could wait for an appeal from either side, and let the game stand if there was no appeal.  But while clueless began the game with another bot using it's processors, clueless wasn't really clueless.  So as TD, I'm intervening without any appeal being lodged.
Title: Re: 2009 Computer Championship
Post by omar on Mar 2nd, 2009, 5:55pm
I just checked the logs for bot_Badger and seems like it may have crashed on move 20g.

This is what I found in a file called x_bot27607.out in the bot's running directory.

Quote:
[badger@gold running]$ more x_bot27607.out
Traceback (most recent call last):
 File "../gtp-bot.py", line 154, in <module>
   sys.exit (BotBridge (engine, '--show-game' in option_dict, *args[1:]).run ())
 File "../gtp-bot.py", line 75, in run
   return self._do_run ()
 File "../gtp-bot.py", line 117, in _do_run
   response = self.engine.command ('genmove', color_to_play).assert_success
 File "/home/badger/gtp.py", line 49, in command
   response = self.maybe_command (*arguments)
 File "/home/badger/gtp.py", line 72, in maybe_command
   raise ValueError ("empty response to command '%s'" % command_line.strip ())
ValueError: empty response to command 'genmove white'


This is what is found in the games log file; 27613.netLog

Quote:
 POST at Mon Mar  2 19:48:28 2009
   auth = 204418483854
   sid = 406041361
   move =
   action = move
 RESPONSE at Mon Mar  2 19:48:29 2009
   error = Gameserver: Bad Move: no steps taken m=[]
Title: Re: 2009 Computer Championship
Post by omar on Mar 2nd, 2009, 9:23pm
The bot_Sharp2008CC vs bot_clueless game has been replayed. Game 98899 replaces game 98863.
Title: Re: 2009 Computer Championship
Post by 99of9 on Mar 2nd, 2009, 10:30pm
Hi Omar,
Unless it makes extra trouble, could I request that you schedule Gnobot's games in either the 3rd or 4th slot of the day?  The first slot means I have to stay up until 1am if I want to see the start.
Thanks
Toby
Title: Re: 2009 Computer Championship
Post by Arimabuff on Mar 3rd, 2009, 4:58am
I don't like the way this is going so far... I mean first Rat times out and then Badger sends an illegal move! I like to see bots lose (or win) because of strategic or tactical reasons not because of a bug.

Hopefully these kinds of incidents won’t happen again…
Title: Re: 2009 Computer Championship
Post by omar on Mar 3rd, 2009, 5:31am

on 03/02/09 at 22:30:13, 99of9 wrote:
Hi Omar,
Unless it makes extra trouble, could I request that you schedule Gnobot's games in either the 3rd or 4th slot of the day?  The first slot means I have to stay up until 1am if I want to see the start.
Thanks
Toby


No problem. I'll set it up for the 4th slot starting tomorrow.
Title: Re: 2009 Computer Championship
Post by omar on Mar 3rd, 2009, 5:35am

on 03/03/09 at 04:58:06, Arimabuff wrote:
I don't like the way this is going so far... I mean first Rat times out and then Badger sends an illegal move! I like to see bots lose (or win) because of strategic or tactical reasons not because of a bug.

Hopefully these kinds of incidents won’t happen again…


Yes, yesterday was really bad. Only one of the four games went smooth. Lets hope things get better.
Title: Re: 2009 Computer Championship
Post by omar on Mar 3rd, 2009, 1:20pm
bot_Rat seems to be going into an infinite loop. The move engine is still continuing to run even now. I will probably have to kill it manually like I had to yesterday. I checked the occ.log file and here is what it has near the end:


Quote:
Applying this move: Rd2n mc2e Eb2e Ra2e

Time for move:    120.0 secs. Maxtime:    656.0 secs.

- - r d - r - -
- - c - d c - -
- r - - - - - -
- - - - - h r r
- - h - - - - -
H r r R e H M r
- R E m D D - R
R - R C C R R R

Goal Search
Depth   Elapsed      Knps    Eval  Main Line
-----  --------  --------  ------  ------------------------------------------
   2       0.0      57.8     206  c8-b8 b8-b7
   3       0.0     835.5     206  c8-b8 h5-h4 b8-b7
   4       0.0   12141.4     208  c8-b8 b8-b7 e7-d7 d7-d6


Looks like it is in the process of generating a move.
Title: Re: 2009 Computer Championship
Post by BlackKnight on Mar 3rd, 2009, 1:46pm

on 03/03/09 at 13:20:30, omar wrote:
I will probably have to kill it manually like I had to yesterday.

Yes, please do it quick and painless! ;)

I'm sorry about this. I think it's the same/similar problem like yesterday.

I get with my local copy (except for what Omar posted):



i.e., nothing, which is "good", because then I can at least fix this bug in the future.
Title: Re: 2009 Computer Championship
Post by omar on Mar 3rd, 2009, 8:16pm

on 03/03/09 at 13:46:45, BlackKnight wrote:
I'm sorry about this. I think it's the same/similar problem like yesterday.


No problem at all. These are all experimental/research bots so unexpected problems are likely to arise.
Title: Re: 2009 Computer Championship
Post by 99of9 on Mar 3rd, 2009, 8:45pm

on 03/03/09 at 05:31:49, omar wrote:
No problem. I'll set it up for the 4th slot starting tomorrow.

Thanks omar.
Title: Re: 2009 Computer Championship
Post by 99of9 on Mar 5th, 2009, 6:30pm
Bomb has opened with exactly the same setup as gold at least 6 times in a row.  This seems quite unlikely when it usually picks from 6 (?) 4 different setups.  Omar, perhaps you could check if some determinism flag has accidentally been set, or if there is some reason that the configuration on the new server is making it behave this way.

Given the book loss to Gnobot, this may even be grounds for an appeal.

EDIT: The info I got was wrong.  It has only used this setup in the two tournament games, plus the aborted game this morning.  Further, it only seems to have 4 opening variants as gold.
Title: Re: 2009 Computer Championship
Post by Fritzlein on Mar 5th, 2009, 7:05pm
It would be a shame if Bomb had been accidentally set to deterministic mode. I'm hoping that the issue is that randomization is on but happens to be poorly implemented, so that the repeat opening setups are the developer's fault.
Title: Re: 2009 Computer Championship
Post by omar on Mar 5th, 2009, 8:26pm
I am afraid it is my fault. There is a bomb.ini file that I forgot to copy over when I was setting up Bomb on the tournament server. The file has 5 fileds and is set like this on the arimaa.com server:

---
30 maximum search depth
600 maximum search time
120 target search time
60 min time to leave
0 random seed, or zero to use time of day
---

If it doesn't find the file Bomb doesn't limit the depth and uses the given time control to decide how much time to spend on it's move. However it defaults to using a non-random setup each time. So as Toby mentioned the game against GnoBot was most effected by this. I've sent an email to the TD about this. We'll wait to see what the decision is.

Thanks for reporting this Toby. Also thanks Karl for the message to check the forum.
Title: Re: 2009 Computer Championship
Post by RonWeasley on Mar 6th, 2009, 4:56am
For it's first two games, I'm going to rule that Bomb's settings are close enough to its tournament configuration that this was an accurate representation of Bomb and the game outcomes stand.  The only difference I see is the maximum search depth being either 30 with the .ini file and unlimited without it.  Judging from the bot developer threads, this limit is rarely reached in the nominal alpha-beta search and a quiescence search would only benefit from this limit being removed.  Note that Badger and clueless do not take advantage of game history the way GnoBot does.

However, the Bomb vs GnoBot game must be replayed to allow Bomb the benefit of its setup randomization.  Thanx and sportsmanship kudos to 99of9 for leading us to the discovery of this mistake.

TD
Title: Re: 2009 Computer Championship
Post by omar on Mar 6th, 2009, 8:59am
Badger timed out in the bot_clueless vs bot_Badger game of round 5. However, when I look at the logs for Badger, it seems that it had sent the move and got an OK response. However something went wrong with the process on the server between accepting the move over the network and saving the move.

I have sent an email to the TD and awaiting a response.
Title: Re: 2009 Computer Championship
Post by omar on Mar 6th, 2009, 9:09am
In trying to trace down what happened on the server,
it seems that the move was received a little too late.
So the process that received the move over the network
could not save it since another process had already
marked the game as ended on time.

I've update the TD about this.
Title: Re: 2009 Computer Championship
Post by doublep on Mar 6th, 2009, 10:12am
I have no problems with this being counted as a loss for Badger.  No chances in this game anyway.  However, if this is not fixed on the server, please remind us about non-trivial lag (i.e. 1 second Badger is configured with is not enough, apparently).

Edit: remind before the next championship, I mean.
Title: Re: 2009 Computer Championship
Post by doublep on Mar 6th, 2009, 10:17am

on 03/02/09 at 17:55:34, omar wrote:
I just checked the logs for bot_Badger and seems like it may have crashed on move 20g. [...]


Yes, that's what it is.  Badger binary crashed and then the interface script raised an exception about empty response.  It is a real bug in the program, but, fortunately, it be should only trigerred when Badger is losing anyway ;)
Title: Re: 2009 Computer Championship
Post by RonWeasley on Mar 6th, 2009, 10:24am
Based on the information above, the result of clueless vs Badger stands with a win for clueless.

TD
Title: Re: 2009 Computer Championship
Post by Fritzlein on Mar 6th, 2009, 1:25pm

on 02/26/09 at 06:40:58, Fritzlein wrote:
40% Clueless
25% Bomb
25% Gnobot
5% Opfor
5% Sharp, Badger, Zombie, Rat

So Bomb is out in 5th place with a 2-3 record, while both OpFor and Badger made the final four!  I guess I was made overly conservative by the fact that I had predicted Bomb's vulnerability in previous years and had been burned.  And who would have believed Badger's meteoric rise?

The consensus was that Fotland wouldn't resume active development as long as Bomb kept winning the Computer Championship, and that proved true.  But will he return now?  If the contest alone isn't enough to entice Fotland, the Challenge money won't help since it remains out of reach and the Computer Championship prize money is minimal.   The only other incentive I can think of is that perhaps the commercialization of Arimaa will provide a big enough market for his software to make further development a good business proposition.
Title: Re: 2009 Computer Championship
Post by omar on Mar 6th, 2009, 3:26pm
Wow, this has been the most amazing and eventful WCC tournament yet. My operational blunders, bots crashing, and some of the most spectacular bot games.

First the Bomb-clueless game in which Bomb was crushed in just 25 moves; then the Bomb-GnoBot game in which Bomb just happened to pick the same setup as the clueless game and lost the same way. Now this last game between Bomb and OpFor was just amazing. Bomb having a small material advantage most of the game and still losing because OpFor had a better understanding of the position. It looked as if Bomb was playing a human opponent. Amazing and a little scary :-)

I hope Fotland will at least make Bomb multi-threaded by next year.
Title: Re: 2009 Computer Championship
Post by omar on Mar 8th, 2009, 7:13am
Yesterday BlackKnight asked me if I could play Rat against Bomb2008CC on the tournament server. I ran a couple games like that and in the first game Rat actually defeated Bomb using the EH attack.

I am also running a couple games with Sharp and Zombie.

If anyone would like to see some other match ups let me know. I'll run them when the servers are idle. Also it will only be done after both bots are out of the tournament.
Title: Re: 2009 Computer Championship
Post by 99of9 on Mar 9th, 2009, 5:55pm

on 03/08/09 at 07:13:39, omar wrote:
If anyone would like to see some other match ups let me know.

More matchups between Badger, Opfor, and Bomb would be very interesting.  All three of them lost to both Clueless and Gnobot, and their third loss was in a cycle to one of the other three (with a win against the other).  So it is not at all clear how they should be pegged relative to one another.  While you have the second server, I'd love to see more of these games.
Title: Re: 2009 Computer Championship
Post by Fritzlein on Mar 9th, 2009, 7:46pm
Yes, there is some ambiguity between places 3rd through 5th.  On the bright side, even without the extra games, it is pretty clear which two bots are the two best, and that's all the tournament format needs to determine.
Title: Re: 2009 Computer Championship
Post by Janzert on Mar 11th, 2009, 1:29pm
I took a look at the network log from the Clueless-Gnobot game this afternoon where Gnobot timed out. Here is the relevant time line of events that I see:

At 16:22:37 the bot interface receives the server gamestate asking for 7b
   Silver reserve time is 96 seconds at the start of the move
At 16:25:59 GnoBot responds with move ef4s ef3n De3e cd3e
   The move is sent to the server by the bot interface
   But the interface either never receives a response from the server or it's a blank response
At 16:26:04 the bot interface once again receives a gamestate asking for 7b
   It shows that 208 seconds of silver's move has elapsed
At 16:29:26 GnoBot once again sends the move ef4s ef3n De3e cd3e
   This time the interface receives a response "ok = 1" as it should
At 16:29:27 the bot interface asks for and receives a gamestate update
   This update shows the timeout result and the interface exits the game

So it appears that there was some sort of server or network error between the arimaa server and CC server.

Janzert
Title: Re: 2009 Computer Championship
Post by RonWeasley on Mar 12th, 2009, 3:59am
Omar confirms the server did not receive GnoBot's move when it should have.  He is going to try to continue the game from the last position with clocks at the correct values.  I have approved this.

TD
Title: Re: 2009 Computer Championship
Post by omar on Mar 12th, 2009, 8:57am
Sorry I forgot to post here in the midst of all the excitement yesterday.

The round 8 clueless vs GnoBot game timed out on move 7b. After checking the logs it appeared that the problem was due to the server. I notified the TD and then logged into the chatroom and was informed that the TD had stopped by and approved continuing the game if it was a server problem.

I had never tried restoring a game and having the bots continue it again, so I wasn't sure if it would work. If it didn't then we would have to replay the game from start.

Well the programs all worked fine and I was able to restore the game and get the bots to continue from 7b. In order to compensate for my time between restoring the game and starting the bots I added 10 seconds to the reserve of each bot from what it was at the start of move 7b. At least that's what I wanted to do; but in actuality I forgot to add 10 secs to GnoBot's reserve and only added it to clueless's reserve. After a few moves we noticed it in the chatroom and decided to stop the bots and retry again, but this time with only 5s added to GnoBots reserve and no time added to clueless's reserve. The reason for this is that as soon as the game is restored GnoBot's time starts ticking down and from experience now I knew it only takes about 5 seconds to get GnoBot started and logged into the game. clueless would get started and logged into the game while it is still GnoBot's turn, so it would not lose any reserve time due to the start up process. This time I got the reserve times right, but the game was restored at move 12b (I think) rather than 7b. We stopped the bots again and tried it a third time. This time everything was set correctly and the game finished to a natural end.
Title: Re: 2009 Computer Championship
Post by Janzert on Mar 12th, 2009, 9:05am
Omar, if you have a chance to run any more non-tournament games on the CC servers and are looking for matchups to try, I would be interested in seeing a few more Badger-OpFor games.

Janzert
Title: Re: 2009 Computer Championship
Post by omar on Mar 12th, 2009, 11:04am

on 03/12/09 at 09:05:52, Janzert wrote:
Omar, if you have a chance to run any more non-tournament games on the CC servers and are looking for matchups to try, I would be interested in seeing a few more Badger-OpFor games.

Janzert


Sure. I'll run one tonight.
Title: Re: 2009 Computer Championship
Post by Janzert on Mar 12th, 2009, 2:27pm
Congratulations to Jdb and Clueless on being the 2009 Computer Championship winner.

Janzert
Title: Re: 2009 Computer Championship
Post by Janzert on Mar 12th, 2009, 2:28pm

on 03/12/09 at 11:04:15, omar wrote:
Sure. I'll run one tonight.


Thanks.

Janzert
Title: Re: 2009 Computer Championship
Post by aaaa on Mar 12th, 2009, 2:35pm
Congratulations to jdb from me as well.
Title: Re: 2009 Computer Championship
Post by RonWeasley on Mar 12th, 2009, 4:34pm
TD congratulations to jdb and clueless.  I also recognize 99of9 and GnoBot for a similarly impressive showing.  These bots really improved this year.  Enough so that people are talking about when a bot might win the Challenge.

Quality of play was quite good from the other bots in this tournament making it what I consider the best WCC so far.  Thanx again to all our bot developers.
Title: Re: 2009 Computer Championship
Post by 99of9 on Mar 12th, 2009, 5:28pm
Yes, well done to all the entrants, and special congrats to jdb.  I think both clueless and gnobot will find winning much harder when opfor and bomb go parallel.  And if badger's rate of improvement continues, it will be knocking over the challenge by next year :).
Title: Re: 2009 Computer Championship
Post by omar on Mar 13th, 2009, 7:39am
Congratulations Jeff (jdb); you've done an increadible job of improving clueless this past year. Congratulations also to all the other developers for making significant progress on the bots. This year reminds me the most of 2004 when the bots Occam and Bomb made such unexpected progress in just one year.
Title: Re: 2009 Computer Championship
Post by mistre on Mar 13th, 2009, 10:59am
Perhaps it is time that Bot_Clueless gets a new name.  I know it was a name created in jest, but to have Arimaa's top rated bot named "Clueless" is a little self-defacing to the great game of Arimaa.

We should have a rename "Clueless" contest...
Title: Re: 2009 Computer Championship
Post by Arimabuff on Mar 13th, 2009, 12:01pm

on 03/13/09 at 10:59:28, mistre wrote:
Perhaps it is time that Bot_Clueless gets a new name.  I know it was a name created in jest, but to have Arimaa's top rated bot named "Clueless" is a little self-defacing to the great game of Arimaa.

We should have a rename "Clueless" contest...

I think it's a terrible idea. It's confusing (just imagine the myriads of newcomers that will have to be explained that what seems to be two different bots are actually the same), It comes at a bad time (I mean you don't change the name of a product when it's at its peak) and also it assumes that the people that play Arimaa are too dumb to appreciate the delightful irony of its name.

Besides, it's always in good taste to be unassuming when you are the greatest (Piece of advice that I promise to follow the day that happens to me). ;D
Title: Re: 2009 Computer Championship
Post by Tuks on Mar 13th, 2009, 2:31pm
maybe...but im not sure it is the same bot, its improvement has been so dramatic since the last CC maybe it should signify this new era of intelligence, dare i say that, with a more appropriate name
Title: Re: 2009 Computer Championship
Post by Fritzlein on Mar 13th, 2009, 2:52pm

on 03/13/09 at 10:59:28, mistre wrote:
We should have a rename "Clueless" contest...

Well, since the "Gno" in GnoBot means "enlightened", i.e. the opposite of clueless, there is an elegant solution.  Whichever bot wins the playoff will thenceforth be called GnoBot, and whichever bot loses will rightfully deserve the name clueless.

Or, better still, we could rank the names of all the Computer Championship entrants from most intimidating to most humble, and then re-assign all the names in order of tournament results.  I think order of intimidation of names is approximately

GnoBot
Badger
sharp
Opfor
Rat
Zombie
bomb
clueless

so the old clueless gets to be Gnobot, the old Gnobot gets to be Badger, etc. on down to the old Rat getting assigned to be clueless.   ;D
Title: Re: 2009 Computer Championship
Post by Fritzlein on Mar 13th, 2009, 3:00pm
OpFor's post-tournament victories over Badger make it seem rather like the "cycle" in the 3rd-4th-5th place finishers was actually a mild fluke, and the true order of strength was

1. clueless
2. GnoBot
3. OpFor
4. bomb
5. Badger

Just by coincidence, that was the order in which they were seeded.  Hmmmm
Title: Re: 2009 Computer Championship
Post by mistre on Mar 13th, 2009, 5:15pm
Looking on this page:

http://arimaa.com/arimaa/mwiki/index.php/Bot_Name_Origins

Clueless - A description of its style of play.

If that is not a good reason for a new name - I don't know what is.
Title: Re: 2009 Computer Championship
Post by Arimabuff on Mar 13th, 2009, 11:05pm

on 03/13/09 at 14:31:02, Tuks wrote:
maybe...but im not sure it is the same bot, its improvement has been so dramatic since the last CC maybe it should signify this new era of intelligence, dare i say that, with a more appropriate name

Maybe they should have renamed the Titanic "Down Under"...  ;D
Title: Re: 2009 Computer Championship
Post by 99of9 on Mar 13th, 2009, 11:48pm

on 03/13/09 at 23:05:45, Arimabuff wrote:
Maybe they should have renamed the Titanic "Down Under"...  ;D

Hey!  No fobbing off overconfident failures on Australia.  :P


Arimaa Forum » Powered by YaBB 1 Gold - SP 1.3.1!
YaBB © 2000-2003. All Rights Reserved.