Arimaa Forum
        (http://arimaa.com/arimaa/forum/cgi/YaBB.cgi)
      

        Arimaa >> Events >> 2011 Arimaa Challenge
        
(Message started by: RonWeasley on Mar 10th, 2011, 12:29pm)
Title: 2011 Arimaa Challenge
Post by RonWeasley on Mar 10th, 2011, 12:29pm
As tournament director, I am pleased to announce the human defenders for the 2011 Arimaa Challenge.

Fritzlein (Karl Junke): a two-time world champion and author of Beginning Arimaa.

rabbits (Gregory Clark): a world championship finalist who defeated multiple world champions in qualifying games.

99of9 (Toby Hudson): a multiple world championship finalist and author of bot_GnoBot.

Alternate Defender: omar (Omar Syed): creator of arimaa.

Please see the Arimaa Challenge site for detailed rules and schedule.  Let's give our support to the defenders, or if you prefer, root for the challenge bot.
Title: Re: 2011 Arimaa Challenge
Post by chessandgo on Mar 10th, 2011, 12:33pm
cool, go humans! And good luck guys! :)
Title: Re: 2011 Arimaa Challenge
Post by lightvector on Mar 10th, 2011, 2:40pm
Yay, good luck everyone!
Title: Re: 2011 Arimaa Challenge
Post by UruramTururam on Mar 10th, 2011, 3:11pm
Good luck defenders!

btw:
I'm a little bit surprised that chessandgo is not selected but perhaps he deserves some rest this year.
Title: Re: 2011 Arimaa Challenge
Post by Adanac on Mar 10th, 2011, 3:35pm

on 03/10/11 at 15:11:27, UruramTururam wrote:
Good luck defenders!

btw:
I'm a little bit surprised that chessandgo is not selected but perhaps he deserves some rest this year.


Omar likes to rotate the defenders each year so that more people get a chance to participate in the Arimaa Challenge.
Title: Re: 2011 Arimaa Challenge
Post by Fritzlein on Mar 10th, 2011, 5:22pm
Yes, rotating the defenders not only shares the work and shares the glory, it also gives a much better sense of how far ahead humans are.  If the team was chessandgo, Adanac, and Fritzlein, and we won the Challenge match 9-0, how much would that tell you?  Rabbits and 99of9 aren't far enough down the ladder to really push the envelope, perhaps because Omar doesn't feel like losing his $10,000, but even so, if they also shut out the Challenger, then we have slightly more information about the level of human dominance.
Title: Re: 2011 Arimaa Challenge
Post by Eltripas on Mar 10th, 2011, 5:31pm
Humans FTW!


on 03/10/11 at 12:29:51, RonWeasley wrote:
rabbits (Gregory Clark): a world championship finalist who defeated multiple world champions in qualifying games.


multiple?
Title: Re: 2011 Arimaa Challenge
Post by aaaa on Mar 10th, 2011, 5:47pm

on 03/10/11 at 17:31:34, Eltripas wrote:
multiple?

Apparently, Ron Weasley used the Time-Turner to witness Adanac winning the championship this year.
Title: Re: 2011 Arimaa Challenge
Post by Arimabuff on Mar 10th, 2011, 7:21pm

on 03/10/11 at 17:47:23, aaaa wrote:
Apparently, Ron Weasley used the Time-Turner to witness Adanac winning the championship this year.

I bet that has to do with Harry Potter.
Title: Re: 2011 Arimaa Challenge
Post by chessandgo on Mar 11th, 2011, 6:25am

on 03/10/11 at 17:47:23, aaaa wrote:
Apparently, Ron Weasley used the Time-Turner to witness Adanac winning the championship this year.


crap, I knew it.
Title: Re: 2011 Arimaa Challenge
Post by Fritzlein on Mar 11th, 2011, 8:46am
The bots are off to a start of five wins and one loss.  So far it looks grim for humanity in the Challenge.  Who can show us some winning patterns?  Have the silicon monsters really gained that much on us since last year?
Title: Re: 2011 Arimaa Challenge
Post by rbarreira on Mar 11th, 2011, 9:02am

on 03/11/11 at 08:46:33, Fritzlein wrote:
The bots are off to a start of five wins and one loss.  So far it looks grim for humanity in the Challenge.  Who can show us some winning patterns?  Have the silicon monsters really gained that much on us since last year?


I don't think it looks that grim... There was one timeout, two games where the person was playing at blitz speed and then two games with people rated much lower than the bots.
Title: Re: 2011 Arimaa Challenge
Post by UruramTururam on Mar 11th, 2011, 9:04am
Don't worry so much I plan to play all 4 allowed games but I don't believe I'll win even once. So don't count points won against me, I did not play last year.  :D also Adanac claims he was playing blitz-like.
Title: Re: 2011 Arimaa Challenge
Post by Fritzlein on Mar 11th, 2011, 9:58am

Quote:
I don't think it looks that grim... There was one timeout, two games where the person was playing at blitz speed and then two games with people rated much lower than the bots.

One thing we'll always be better at than computers is making excuses.  ;)  Not that I disbelieve the explanations for our collective slow start: I see the ratings; I see Adanac's thinking time per move; I see that there was still play left when Arimabuff timed out.  But the great thing about winning is then you don't have to explain why you won. :)

Year  Pairs  Decisive  Winner / Score / Perf  Loser / Score / Perf
----  -----  --------  ---------------------  --------------------
2007     12    .    2    . bomb / 2 / 2087    . Zombie / 0 / 1876  
2008     16    .    7    . bomb / 6 / 1918    .  sharp / 1 / 1576
2009     23    .    7  clueless / 5 / 1910    . GnoBot / 2 / 1792
2010     25    .   11    marwin / 6 / 2065    clueless / 5 / 1960

Yes, I know that performance rating is a very noisy metric when there have only been a few games played, but the bots 5-1 record so far means they are performing at a rating of 2443! :o
Title: Re: 2011 Arimaa Challenge
Post by Fritzlein on Mar 11th, 2011, 10:00am

on 03/11/11 at 09:04:58, UruramTururam wrote:
Don't worry so much I plan to play all 4 allowed games but I don't believe I'll win even once.

Thanks for playing in the screening despite the odds against you.  You might surprise yourself and find a winning pattern.  Every scrap of information helps the defenders, and leaves us less blind going into the Challenge games.
Title: Re: 2011 Arimaa Challenge
Post by UruramTururam on Mar 11th, 2011, 10:13am

on 03/11/11 at 09:58:11, Fritzlein wrote:
One thing we'll always be better at than computers is making excuses.  ;)  


I remember playing computer version of Magic: the Gathering. after losing a game the computer opponent always made a witty comment like: "Yeah, you won. Of course I did not draw enough lands..."
Title: Re: 2011 Arimaa Challenge
Post by Fritzlein on Mar 11th, 2011, 10:33am

on 03/11/11 at 10:13:19, UruramTururam wrote:
I remember playing computer version of Magic: the Gathering. after losing a game the computer opponent always made a witty comment like: "Yeah, you won. Of course I did not draw enough lands..."

Heheh, that's not the bot making an excuse; that's the developer making fun of humans (like me) who always blame their losses on luck.  :)
Title: Re: 2011 Arimaa Challenge
Post by UruramTururam on Mar 11th, 2011, 10:38am
Yet so far programs do what their developers wanted ordered them to do...

Btw: look here - http://arimaa.com/arimaa/gameroom/comments.cgi?gid=175288 - s**t happens to bots too.
Title: Re: 2011 Arimaa Challenge
Post by rbarreira on Mar 11th, 2011, 12:24pm
Now that Tuks has lost against sharp, I think you can start saying it looks grim a bit more legitimately :)
Title: Re: 2011 Arimaa Challenge
Post by Fritzlein on Mar 11th, 2011, 12:35pm

Quote:
Yes, I know that performance rating is a very noisy metric when there have only been a few games played, but the bots 5-1 record so far means they are performing at a rating of 2443!

Make that 6-1, for a performance of 2464.  By a remarkable coincidence, my game room rating is 2464.  In fairness, though, we haven't yet heard Tuks's reason for losing.  ;)
Title: Re: 2011 Arimaa Challenge
Post by UruramTururam on Mar 11th, 2011, 1:16pm

on 03/11/11 at 12:35:03, Fritzlein wrote:
we haven't yet heard Tuks's reason for losing.  ;)


Here it goes (taken from the game comment):


Quote:
sharp is good. Good news for everyone though, sharp is open to bait and tackle :) shame i'm an idiot and after setting up the position i messed up the execution.

also missed the false protection at the end and there bunch of other mistakes i don't care to reexamine
Tuks     Fri 20:01 YLT
Title: Re: 2011 Arimaa Challenge
Post by Adanac on Mar 11th, 2011, 1:51pm

on 03/11/11 at 12:35:03, Fritzlein wrote:
Make that 6-1, for a performance of 2464.  By a remarkable coincidence, my game room rating is 2464.  In fairness, though, we haven't yet heard Tuks's reason for losing.  ;)


Bots are improving at the rate of about 400 points per year.  So they'll be invincible by 2013   :o
Title: Re: 2011 Arimaa Challenge
Post by mistre on Mar 11th, 2011, 3:42pm

on 03/11/11 at 12:35:03, Fritzlein wrote:
Make that 6-1, for a performance of 2464.  By a remarkable coincidence, my game room rating is 2464.  In fairness, though, we haven't yet heard Tuks's reason for losing.  ;)


What is even more scary is how badly Bot_Quad crushed Sharp!
Title: Re: 2011 Arimaa Challenge
Post by chessandgo on Mar 11th, 2011, 4:19pm

on 03/11/11 at 15:42:42, mistre wrote:
What is even more scary is how badly Bot_Quad crushed Sharp!


Yeah this, that was an impressive game. I must admit I had pretty much given up on the whole institution of playing bots, but it seems they've had a big leaf in playing strength this year, I'll get a shot at marwin tomorrow hopefully. Also it's kinda frightening to wonder about sharp with a better treatment of advanced rabbits (possibly next year...).
Title: Re: 2011 Arimaa Challenge
Post by mistre on Mar 11th, 2011, 4:26pm
Good game Jean vs Sharp, your superior goal attack understanding made the difference.
Title: Re: 2011 Arimaa Challenge
Post by Fritzlein on Mar 11th, 2011, 4:54pm
A commanding victory, Jean, the more impressive for winning (at least as I perceive it) against sharp's strength rather than against sharp's weakness.  If it turns out that people win against sharp by swarming and getting in a goal race and lose when they play a home game and take a camel hostage, I will have to re-adjust my expectations.

Now that sharp has a loss, I can give each bot an independent performance rating.  I will maintain the table by editing this post rather than by re-posting every time I re-calculate.

For the record, I was expecting only modest improvement in marwin, i.e. performance in the 2050-2100 range, and sharp to perform perhaps worse against humans, say in the 1950-2050 range despite beating marwin head-to-head.  It seems I already have to kiss that hope goodbye; now the question is simply how worried I need to be, and whether there is any easy pattern for winning against each.  Please, everyone take a shot at these bots; surely they have exploitable weaknesses if only we can discover them.

Year  Pairs  Decisive  Winner / Score / Perf  Loser / Score / Perf
----  -----  --------  ---------------------  --------------------
2007     12    .    2    . bomb / 2 / 2087    . Zombie / 0 / 1876  
2008     16    .    7    . bomb / 6 / 1918    .  sharp / 1 / 1576
2009     23    .    7  clueless / 5 / 1910    . GnoBot / 2 / 1792
2010     25    .   11    marwin / 6 / 2065    clueless / 5 / 1960
2011     39    .   11     sharp / 5 / 2102    . marwin / 6 / 2110
Title: Re: 2011 Arimaa Challenge
Post by rbarreira on Mar 11th, 2011, 4:59pm
How did sharp's performance rating go from 2464 to 2521 after it lost a game?
Title: Re: 2011 Arimaa Challenge
Post by chessandgo on Mar 11th, 2011, 4:59pm
Thanks guys.
Title: Re: 2011 Arimaa Challenge
Post by chessandgo on Mar 11th, 2011, 5:00pm

on 03/11/11 at 16:59:21, rbarreira wrote:
How did sharp's performance rating go from 2464 to 2521 after it lost a game?


Elo performance does that sometimes, when you lose to a high rated player after winning against several other players, I think (?).
Title: Re: 2011 Arimaa Challenge
Post by Fritzlein on Mar 11th, 2011, 5:56pm

on 03/11/11 at 16:59:21, rbarreira wrote:
How did sharp's performance rating go from 2464 to 2521 after it lost a game?



on 03/11/11 at 17:00:57, chessandgo wrote:
Elo performance does that sometimes, when you lose to a high rated player after winning against several other players, I think (?).

There are two standard ways to calculate performance rating.  One way has the disadvantage that a loss to a much-higher-rated player can help you and a win over a much-lower-rated player can hurt you.  I don't like this feature, so I use the other way.  I say performance is the rating that would have predicted the score you got.

The disadvantage of the way I like is that if you have no losses, your performance is plus-infinity and if you have no wins, your performance rating is minus-infinity.  This is because even a very high rating doesn't predict you will never lose.  This was a problem when sharp had no losses, so I was lumping sharp and marwin together to get a combined performance of 2464.  If I had split them apart, marwin's performance would have been 2230 and sharp's would have been plus infinity.  When sharp lost to chessandgo, its performance rating dropped from plus infinity to only 2521.

That's weird, I know, but I like it better than the other disadvantage.  For example, sharp just now beat 722caasi, rated 1804.  In my calculation, that boosts sharp's performance from 2521 to 2526, a gain of five points.  In the other calculation, beating 722caasi would have lowered sharp's performance from 2450 to 2409, a loss of forty-one points.  Since I can't stand the illogical of losing points for a win, I tolerate the infinities.  :)

In a footnote, to calculate these performance ratings, I am making no reference to WHR; I'm simply use the opponent's game room rating at the time of the game.  I know game room ratings are inaccurate, but I put up with it.  I will, however, draw the line at using hanzack's game room rating, should he participate in the screening phase.  Instead I will use his performance from the World Championship, where his 4-4 record against his strong schedule would have been exactly predicted for someone with a rating of 2060.
Title: Re: 2011 Arimaa Challenge
Post by rbarreira on Mar 11th, 2011, 6:01pm
Oh, that makes sense... I wrongly assumed 2464 was the previous rating for sharp. Thanks for the explanation!
Title: Re: 2011 Arimaa Challenge
Post by Fritzlein on Mar 11th, 2011, 10:10pm
Thanks, 722caasi, for being the first one to complete all four games of the screening.  I was very impressed with your last game against sharp; you had a comfortable material advantage and a solid position as well.  Thanks for showing us that sharp is beatable.
Title: Re: 2011 Arimaa Challenge
Post by 722caasi on Mar 11th, 2011, 10:19pm
Thanks for ignoring the ends of my games :)
Sharp isn't too hard to beat if you don't get swarmed and you can play an endgame. Best way to not get swarmed? Tie down their elephant. Thus, give horse hostages.
Title: Re: 2011 Arimaa Challenge
Post by Fritzlein on Mar 11th, 2011, 10:25pm

on 03/11/11 at 22:19:36, 722caasi wrote:
Best way to not get swarmed? Tie down their elephant. Thus, give horse hostages.

This could be a very valuable tip.  I'll be watching further games to see how this pans out.
Title: Re: 2011 Arimaa Challenge
Post by chessandgo on Mar 12th, 2011, 7:53am

on 03/11/11 at 16:19:49, chessandgo wrote:
I'll get a shot at marwin tomorrow hopefully.


I meant at quad and/or at marwin. Marwin, check :) Based on that sole game, I'm not impressed by marwin's trap control (and not by its treatment of advanced rabbits either, like sharp). Is bot_quad available to play only when aaaa runs it?
Title: Re: 2011 Arimaa Challenge
Post by Fritzlein on Mar 13th, 2011, 10:16pm
The_Jeh's win over marwin increases sharp's lead to 2.5.  The drama of the screening has changed from "which bot will win?" to "can anyone beat sharp?"  (besides chessandgo, I mean)
Title: Re: 2011 Arimaa Challenge
Post by Hippo on Mar 14th, 2011, 2:39am
I hope there are still players whose have a chance. But sharp looked like easily be in top 10. I hope Tuks and Adanac would concentrate better for the other try.

Wow good news for humanity ... Sharp totally lost in the ending with aaaa.
Title: Re: 2011 Arimaa Challenge
Post by Fritzlein on Mar 14th, 2011, 6:44pm

on 03/13/11 at 22:16:56, Fritzlein wrote:
The drama of the screening has changed from "which bot will win?" to "can anyone beat sharp?"

Just as I say it, it becomes untrue.  Aaaa's win over sharp cuts sharp's lead to 1.5.  Plus Manuel's win over sharp and Hippo's loss to marwin in uncompleted pairs have narrowed the gap in performance between the two bots to just 2274 to 2158 in sharp's favor.  This may yet be a race for which bot qualifies, and the Arimaa Challenge might not yet be in danger.
Title: Re: 2011 Arimaa Challenge
Post by foggy on Mar 15th, 2011, 3:27am
What is the link for screening results? Previously I used the link "Play the screening bots" from announcements section, but that link is not always available (and is not available now). It would be good to have a stable link for results (maybe in coverage or tournament description pages).

Edit: Now announcements section is updated, so I can answer my question about the link: http://arimaa.com/arimaa/challenge/2011/showBestBot.cgi. And the new version of the page is great - track all screening games and comments in 1 place (thanks!).
Title: Re: 2011 Arimaa Challenge
Post by Fritzlein on Mar 15th, 2011, 1:44pm
Omar, I second foggy's thanks for listing all the screening games in one place.  That makes it easy to catch up on what I missed, even though nobody was chatting about it in the chat room today.

Tuks victory over marwin puts sharp's lead back at 2.5.  I'm thrilled with the broad participation in this year's screening.  At this rate it seems possible we will surpass even last year's record participation.
Title: Re: 2011 Arimaa Challenge
Post by Fritzlein on Mar 18th, 2011, 12:38am
The plot thickens.  Marwin's victory over Harren cuts sharp's lead back to only 1.5.  In addition, marwin has pulled ahead in performance rating, 2128 to 2104.  This reflects marwin's victories in several incomplete pairs.  If all pairs are completed at the level of performance so far, marwin will probably pull ahead.
Title: Re: 2011 Arimaa Challenge
Post by SpeedRazor on Mar 18th, 2011, 3:00pm
If a computer won the Arimaa challenge it would be huge!  Newsworthy!  But...

Would people actually take it seriously if the absolute best Human players weren't represented?  I'm thinking about that huge schism/fiasco in the chess world, where the Official World Chess Champion from the Official World Chess Regulating Body, FIDE, was not considered the actual best chess player in the world by virtually every chess player and media organization, because Gary Weinstein (Kasparov) didn't compete in it.

I'd hate to have to see anybody lose $10,000 twice, because the first time it seemed tainted...

• Note:  please forgive me if I don't understand the Arimaa challenge rules well enough •
Title: Re: 2011 Arimaa Challenge
Post by Adanac on Mar 18th, 2011, 4:43pm

on 03/18/11 at 15:00:55, SpeedRazor wrote:
If a computer won the Arimaa challenge it would be huge!  Newsworthy!  But...

Would people actually take it seriously if the absolute best Human players weren't represented?  I'm thinking about that huge schism/fiasco in the chess world, where the Official World Chess Champion from the Official World Chess Regulating Body, FIDE, was not considered the actual best chess player in the world by virtually every chess player and media organization, because Gary Weinstein (Kasparov) didn't compete in it.

I'd hate to have to see anybody lose $10,000 twice, because the first time it seemed tainted...

• Note:  please forgive me if I don't understand the Arimaa challenge rules well enough •


The $10,000 prize will only be won once, not twice, but yes it will be a shame if the #1 rated player does not participate in the year that the prize is claimed by  a bot developer.  Even if the # 1, 2 & 3 ranked players are all defeated you could get different criticisms if the # 4, 5 & 6 players were a combined 6-0 against the winning bot in the screening round (it’s not clearly better than all the best humans after all?)  But, in both cases, I would consider that a legitimate victory by the bot developer against steep odds.  If the prize is claimed before 2020 under any circumstances then I’d say that the developers have gone a lot further in 15 years than I ever expected they would when I joined the Arimaa community in 2005.  It was only two years ago that a bot lost a horse handicap game in the Challenge Match, for example.
Title: Re: 2011 Arimaa Challenge
Post by SpeedRazor on Mar 18th, 2011, 5:05pm
Thanks very much Adanac, all of your arguments make sense to me  :)
Title: Re: 2011 Arimaa Challenge
Post by Arimabuff on Mar 20th, 2011, 10:32am
It looks like there's hope for Marwin yet...
Title: Re: 2011 Arimaa Challenge
Post by Hippo on Mar 21st, 2011, 5:26am
And now marwin leads by 1,5 point.
Title: Re: 2011 Arimaa Challenge
Post by Arimabuff on Mar 21st, 2011, 10:31am

on 03/21/11 at 05:26:07, Hippo wrote:
And now marwin leads by 1,5 point.

It's strange how things turn out.
Title: Re: 2011 Arimaa Challenge
Post by UruramTururam on Mar 21st, 2011, 11:35am
Hm, that means that my discovery of time stealing opening against Marvin may be not a totally useless one.  ;)
Title: Re: 2011 Arimaa Challenge
Post by mistre on Mar 23rd, 2011, 7:26am
Marwin's lead is cut to 1.  If one of piv, sconibulus, or camelback can beat Marwin, then it will be tied.

What happens if its tied?  Also, has the 2nd place bot in the computer championship ever won out in the screening?
Title: Re: 2011 Arimaa Challenge
Post by UruramTururam on Mar 23rd, 2011, 7:34am
If tied - the champion wins.

And no, this time could be the first one that the computer champion and the challenge competitor are different programs.
Title: Re: 2011 Arimaa Challenge
Post by aaaa on Mar 25th, 2011, 11:40am
Omar, could you change the layout of the bot scores such that a player gets a separate row for each color? That way, one could spot more quickly whether and with which counterpart a screening game is paired.
Title: Re: 2011 Arimaa Challenge
Post by omar on Mar 28th, 2011, 1:05pm
Try this: http://arimaa.com/arimaa/challenge/2011/showBestBot3.cgi
Title: Re: 2011 Arimaa Challenge
Post by aaaa on Mar 28th, 2011, 4:07pm
I rather had something like the following in mind:


Code:
<html>
<body>
<center>
<table border=1>
<tr>
<th rowspan="2">Player</th>
<th rowspan="2">Color</th>
<th colspan="2">bot_sharp</th>
<th colspan="2">bot_marwin</th>
</tr>
<tr>
<th>Games</th>
<th>Points</th>
<th>Games</th>
<th>Points</th>
</tr>
<td rowspan="3">rbarreira</p>
<th>Gold</th>
<td><font color=red>*</font> <a target=_blank href="/arimaa/gameroom/comments.cgi?gid=175606">r 22</a></td>
<td>1</td>
<td><font color=red>*</font> <a target=_blank href="/arimaa/gameroom/comments.cgi?gid=175632">r 20</a></td>
<td>1</td>
</tr>
<tr>
<th>Silver</th>
<td><font color=red></font> <a target=_blank href="/arimaa/gameroom/comments.cgi?gid="> </a></td>
<td></td>
<td><font color=red>*</font> <a target=_blank href="/arimaa/gameroom/comments.cgi?gid=175920">g 26</a></td>
<td><s>1</s></td>
</tr>
<tr>
<th>Both</th>
<td></td>
<td>1</td>
<td></td>
<td>1</td>
</tr>
</table>
</center>
</body>
</html>
Title: Re: 2011 Arimaa Challenge
Post by Fritzlein on Mar 29th, 2011, 12:29am
With time nearly out on the screening, Marwin has a 0.5 point lead, but in all four unfinished pairs (ngerhart, camelback, piv, Sconibulus) sharp has a win while marwin has yet to be played.  It would be a shame if the screening finished in this state, given that marwin has a fair chance of losing one of those four if all pairs are completed.  Can any of you find time to finish up these pairs in the next couple of days?

[EDIT]
Oh, I see rbarriera also has an unfinished pair, with mawin having a win and sharp uplayed.  That makes it slightly less lopsided, but it would still be great to see these pairs finished out.
Title: Re: 2011 Arimaa Challenge
Post by rbarreira on Mar 29th, 2011, 2:25am
I plan to play my game, although I'm quite unlikely to beat sharp... Maybe someone should post game comments asking those other players if they can play another game.
Title: Re: 2011 Arimaa Challenge
Post by Fritzlein on Mar 30th, 2011, 2:58pm
Great, rbarriera.  Thanks for finishing off your pair.  Of course it is impossible for such a chaotic selection process to be entirely fair, but at least that will be one less "what might have been?" hovering around the outcome.
Title: Re: 2011 Arimaa Challenge
Post by UruramTururam on Mar 31st, 2011, 2:50am
We have an interesting situation now:
  • The computer world champion, sharp, is likely not to win the Challenge qualification.
  • Sharp seemed almost unbeatable at first and it had a substantial advantage at the beginning of screening period and then that advantage melted to less then nothing.

I think that it could explained by fundamentally different approaches sharp and marwin use to calculate their moves. Marwin represents a classical "search & evaluate" approach. Sharp uses a few clever heuristics to narrow its space of the moves of interest at very early stages of search.

What this is likely to cause: sharp plays more brilliantly. Just recall the "elephant in the hole" move during the crucial phase of its match against marwin during the championship. This approach is likely to take this bot opponents by surprise. But it leaves some space for weaknesses to explore: some tricks that mislead sharp's heuristics. When they are spotted sharp can be overrun by repetitive use of these tactics, while marwin still requires just careful strategical playing against it.

That may lead to a paradox that a human-like machine can beat machine-like one, but it's still not smart enough to fool well playing human opponents.
Title: Re: 2011 Arimaa Challenge
Post by megajester on Mar 31st, 2011, 5:16am

on 03/31/11 at 02:50:44, UruramTururam wrote:
We have an interesting situation now:
  • That may lead to a paradox that a human-like machine can beat machine-like one, but it's still not smart enough to fool well playing human opponents.

Well, that would seem to validate the Arimaa Challenge as a catalyst for genuine advances in AI. The future looks bright! We have 8 more years of titanic man-machine struggle to come...
Title: Re: 2011 Arimaa Challenge
Post by lightvector on Mar 31st, 2011, 5:40am
Actually, sharp is reasonably full-width. Any move is reduced in depth by at most 2 (and if it looks good, it gets re-searched at full depth again).

I maintain that sharp is really strategically weak. It just is reasonably good at tactics when the board gets really messy, and has one good trick. Maybe it's also a bad trick, I guess: give up a camel hostage and swarm a lot.

;D

It turns out that being strategically weak is okay against other bots if you know how to do okay in messy fights, (although it's getting less and less okay), because other bots don't know how to consistently play "solidly" and punish strategic mistakes, and they make many as well. So my focus has been on making the eval tactically strong, because that's what beats other bots.

The problems are the usual ones for any traditional bot, namely that I didn't have any time to tune the "camel is afraid of elephant" evaluation (so sharp gives up camel hostages easily), and I didn't have time to tune the advanced rabbit evaluation (spent only a day writing the code, haven't revisited it since), etc. That is, it's a typical evaluation function with lots of holes.

Lots of work for next year though!  :)
Title: Re: 2011 Arimaa Challenge
Post by rbarreira on Mar 31st, 2011, 2:50pm
What is the actual screening period? In the 2011 challenge page it says "Mar 17, 2011 to Mar 29, 2011 ", but it started before that and there were at least two screening games after that, so the server might be still allowing screening games.

edit - oh, in the gameroom announcements it says "March 10 to March 31".
Title: Re: 2011 Arimaa Challenge
Post by Arimabuff on Apr 1st, 2011, 7:52am
Does that mean that Sharp is a bot's bot?
Title: Re: 2011 Arimaa Challenge
Post by aaaa on Apr 1st, 2011, 10:53am
Sharp really got unlucky here as all five uncounted games are wins of its, undoubtedly giving it a better performance rating than Marwin.
Title: Re: 2011 Arimaa Challenge
Post by Hippo on Apr 1st, 2011, 11:36am
Unfortunately I have spent all last nights making tax report ... . I hope these bots would be around for several more days ...
Title: Re: 2011 Arimaa Challenge
Post by omar on Apr 1st, 2011, 12:33pm
Yes, too bad some of the players did not get to finish their game against marwin. It definitely was unlucky for sharp. The first bot you play against is picked randomly, so it's just an unlucky coincidence that sharp was picked as the first bot against opponents that did not continue playing more games.

Both bot are very good and this is perhaps the closest screening period I've seen. The defenders will have to very  carefully this year.

Thanks everyone who played against the screening bots. Your games will be very helpful to the defenders.
Title: Re: 2011 Arimaa Challenge
Post by chessandgo on Apr 2nd, 2011, 3:48am

on 04/01/11 at 11:36:30, Hippo wrote:
Unfortunately I have spent all last nights making tax report ...


Won too much in the spectator contest? :)
Title: Re: 2011 Arimaa Challenge
Post by Fritzlein on Apr 3rd, 2011, 11:46pm
I have updated the table with the final performance ratings.

Year  Pairs  Decisive  Winner / Score / Perf  Loser / Score / Perf
----  -----  --------  ---------------------  --------------------
2007     12    .    2    . bomb / 2 / 2087    . Zombie / 0 / 1876  
2008     16    .    7    . bomb / 6 / 1918    .  sharp / 1 / 1576
2009     23    .    7  clueless / 5 / 1910    . GnoBot / 2 / 1792
2010     25    .   11    marwin / 6 / 2065    clueless / 5 / 1960
2011     40    .   11    marwin / 6 / 2110    .  sharp / 5 / 2109

It is odd that marwin at 27-13 would have a higher performance rating than sharp at 31-14.  For winning 27 instead of 26 out of the 40 pairs that counted, marwin gained an extra 29 Elo points.  For winning 5 unpaired games that didn't count (Sconibulus, camelback, piv, negerhart, Eltripas), sharp gained an extra 32 Elo points.  So why didn't sharp win the performance rating battle by three points instead of losing by one point?

Alas, the swing of four Elo was caused by the random fluctuations of gameroom ratings.  Every player's rating changes from game to game, so the bots get slightly different credit for playing the same opponents.  That smidgin of randomness is a reminder to be cautious about reading too much into the performance ratings.

Nevertheless, although uncertainty bedevils all statistics, I feel in my gut that the bots are truly stronger in 2011 than they were in 2010.  I was hopeful that humans had stolen a march on bots this year, and that our advantage had perhaps even widened, but I no longer believe that.  I am now afraid that the tide has turned for good, and the gap will shrink yearly until humanity can no longer hold out.  The only question is whether the lines will cross before 2020 or after.

On the other hand, comparing marwin's performance rating of 2110 to the WHR ratings, there are still nineteen humans rated higher, thirteen of them active.  The growth of the Arimaa community and the improvement of established members is a reason to hope.  As long as we keep having an infusion of fresh energy and ideas, we just might deepen our own strategic understanding beyond what the bots can keep up with.

I am grateful to everyone who participated in the screening this year.  Forty completed pairs shattered the old record of twenty-five.  That's tremendous participation.  I don't think marwin changed much from last year, but even if it had, its thirteen losses in the screening would give me plenty to study while preparing to defend the Challenge.  Thanks, guys!
Title: Re: 2011 Arimaa Challenge
Post by chessandgo on Apr 4th, 2011, 4:57am
Oh yes, last year I felt like the bots had already improved by a decent amount, but this year's improvement appears  even more spectacular. I'm not that pessimistic wrt the challenge though, maybe the bots can improve just as much next year, but I'd be very surprised if they can keep up this pace for several years. Well, we sure have many over-skilled developers around, so maybe they'll prove me wrong :)

Good luck for the challenge games!
Title: Re: 2011 Arimaa Challenge
Post by Fritzlein on Apr 9th, 2011, 3:34pm
We should start keeping audience records for this event too.  There were 34 peak listeners for Arimabuff and omar commentating on my game against marwin.  It's a great compliment to the commentators to keep an audience engaged for more than two and a half hours, especially with the position in "mop up mode" for so long.  Before this year's World Championship, an audience of 34 would have been a record turnout for any event; now it has to play second fiddle to a record of 44 simultaneous listeners and 52 logins.  Still, that is a fantastic turnout for the man vs. machine match which (we hope) should be lopsided in humanity's favor, and a far larger audience than for any previous Arimaa Challenge.
Title: Re: 2011 Arimaa Challenge
Post by omar on Apr 9th, 2011, 6:13pm
I think we had a peak of about 27 during Toby's game; which is very good for an early morning game.

I didn't get much chance to post links to the live game on other forums. This probably would have increased the listeners by at least 10. I usually post to the the appropriate section of the forum on sites like BoardGameGeek.com, chess.com, LittleGolem.net, etc.

I found that posting the link directly to the game window near the start of the game works best. If you know of other forums we can post to, please help out. I've posted a short link to the next game in the commentary spreadsheet.
Title: Re: 2011 Arimaa Challenge
Post by Fritzlein on Apr 9th, 2011, 6:33pm

on 04/09/11 at 18:13:19, omar wrote:
I've posted a short link to the next game in the commentary spreadsheet.

I just found out that the ".ly" in http://bit.ly/e7ZR4b stands for Libya.  I wonder if these URLs will disappear because nobody wants to be even remotely associated with Qaddafi.


on 04/09/11 at 18:13:19, omar wrote:
I didn't get much chance to post links to the live game on other forums. This probably would have increased the listeners by at least 10. I usually post to the the appropriate section of the forum on sites like BoardGameGeek.com, chess.com, LittleGolem.net, etc.

Wow, now I am even more impressed with the high turnout.  I had assumed it was due at least in part to active advertising.  I guess the live commentary and the man vs. machine clash is a hotter commodity than I thought.  Apparently the games don't have to be dramatic (and I hope humans win all nine in boring fashion) as long as the announcers find something interesting to say.

Knowing the potential size of the audience makes me want to jump in and help The_Jeh tomorrow.  I hope it's OK with him if I flit in and out of the commentary booth on short notice.
Title: Re: 2011 Arimaa Challenge
Post by The_Jeh on Apr 9th, 2011, 11:16pm

on 04/09/11 at 18:33:31, Fritzlein wrote:
Knowing the potential size of the audience makes me want to jump in and help The_Jeh tomorrow.  I hope it's OK with him if I flit in and out of the commentary booth on short notice.


Like I said, I would appreciate someone else there, too. It's hard to keep talking at 2 minutes/move.
Title: Re: 2011 Arimaa Challenge
Post by Fritzlein on Apr 9th, 2011, 11:25pm

on 04/09/11 at 23:16:57, The_Jeh wrote:
Like I said, I would appreciate someone else there, too. It's hard to keep talking at 2 minutes/move.

Great, I'll try to be there with you, hopefully for the whole game.
Title: Re: 2011 Arimaa Challenge
Post by Arimabuff on Apr 10th, 2011, 12:13am
I may drop by but I won't intervene unless asked to, given the presence of Fritz there already.
Title: Re: 2011 Arimaa Challenge
Post by Fritzlein on Apr 10th, 2011, 3:12pm

on 04/09/11 at 15:34:32, Fritzlein wrote:
There were 34 peak listeners for Arimabuff and omar commentating on my game against marwin.

It didn't take long to break that record.  We had a peak of 40 listeners for The_Jeh, Arimabuff, and myself commentating on rabbits victory over marwin.  Thanks, omar, for advertising and getting the radio working and recording and uploading the capture.  What fun!
Title: Re: 2011 Arimaa Challenge
Post by omar on Apr 10th, 2011, 3:27pm

on 04/10/11 at 15:12:33, Fritzlein wrote:
It didn't take long to break that record.  We had a peak of 40 listeners for The_Jeh, Arimabuff, and myself commentating on rabbits victory over marwin.  Thanks, omar, for advertising and getting the radio working and recording and uploading the capture.  What fun!


Thanks for joining the commentary. It always makes the game more enjoyable listening to your commentary.
Title: Re: 2011 Arimaa Challenge
Post by omar on Apr 10th, 2011, 3:28pm
I've uploaded the videos for the game here:

http://arimaa.com/arimaa/videos/2011ac/
Title: Re: 2011 Arimaa Challenge
Post by megajester on Apr 10th, 2011, 3:55pm
I will definitely be having a listen.
Title: Re: 2011 Arimaa Challenge
Post by omar on Apr 10th, 2011, 4:57pm
I tried to post about the challenge match game to some forums. Here is an example:
http://www.talkchess.com/forum/viewtopic.php?p=402590

Basically I start a thread about the 2011 challenge match with a link to the showGames page:
http://arimaa.com/arimaa/challenge/2011/showGames.cgi

Then at the start of the game, I post a reply to the thread with a short link to the game in progress.

If you would like to post about the Arimaa challenge match to other forums that you visit, please try to use this approach and be sure to post in the appropriate section of the forum; sometime the only appropriate section might be the 'Off Topic' section. You can find the short link to the game in the shared spreadsheet under the announcements.

Also if you have a twitter account please try to send out a message about the Arimaa challenge match game near the beginning of the game. Here are some examples:
http://twitter.com/#!/search/arimaa

I think with a little more promotion we should be able to hit 50+ peak listeners.
Title: Re: 2011 Arimaa Challenge
Post by Fritzlein on Apr 19th, 2011, 2:20am
Omar, I changed my time slot to 64 for the final round, but you still scheduled me for slot 66.  Would it be possible to move my game two hours earlier?  Thanks in advance!
Title: Re: 2011 Arimaa Challenge
Post by omar on Apr 20th, 2011, 2:06pm
Got it. Thanks for bringing it to my attention. I hope Patrick notices the change also.
Title: Re: 2011 Arimaa Challenge
Post by Fritzlein on Apr 20th, 2011, 2:32pm

on 04/20/11 at 14:06:31, omar wrote:
Got it. Thanks for bringing it to my attention. I hope Patrick notices the change also.

Thanks for fixing this!  (I changed my time preference after it turned out that my Challenge game last week conflicted with a pickup Frisbee game nearby.  We'll see if there is pickup at the same time this week...)
Title: Re: 2011 Arimaa Challenge
Post by Arimabuff on Apr 21st, 2011, 2:34am

on 04/20/11 at 14:06:31, omar wrote:
Got it. Thanks for bringing it to my attention. I hope Patrick notices the change also.

I did. :)
Title: Re: 2011 Arimaa Challenge
Post by omar on Apr 21st, 2011, 6:23pm
Actually I forgot to put the time of rabbits game on my calendar and didn't get a reminder. I missed the start of the game until Janzert called and reminded me.

Sharp was also running on the server at the same time. I called the TD and he ruled that we should halt the game. It was found that right from the start of rabbits game Sharp was running (game 180690):

http://gold.arimaa.com/logs/20110421/21/0401

So I have reset the game to start again at the time requested by rabbits.

Sorry about the inconvenience this has caused.
Title: Re: 2011 Arimaa Challenge
Post by Arimabuff on Apr 21st, 2011, 10:41pm

on 04/21/11 at 18:23:50, omar wrote:
Actually I forgot to put the time of rabbits game on my calendar and didn't get a reminder. I missed the start of the game until Janzert called and reminded me.

Sharp was also running on the server at the same time. I called the TD and he ruled that we should halt the game. It was found that right from the start of rabbits game Sharp was running (game 180690):

http://gold.arimaa.com/logs/20110421/21/0401

So I have reset the game to start again at the time requested by rabbits.

Sorry about the inconvenience this has caused.

I'll be there; hopefully more people will have caught on than last time.
Title: Re: 2011 Arimaa Challenge
Post by UruramTururam on Apr 22nd, 2011, 12:59am
Heh I wonder if rabbits will try what the listeners suggested last time - to win a handicapped game: without letting any rabbit being captured.  ;)
Title: Re: 2011 Arimaa Challenge
Post by Arimabuff on Apr 22nd, 2011, 5:32am

on 04/22/11 at 00:59:51, UruramTururam wrote:
Heh I wonder if rabbits will try what the listeners suggested last time - to win a handicapped game: without letting any rabbit being captured.  ;)

That wouldn't be new as Fritz already won a no piece taken game.


Arimaa Forum » Powered by YaBB 1 Gold - SP 1.3.1!
YaBB © 2000-2003. All Rights Reserved.