Arimaa Forum - Print Page


    
      
        Arimaa Forum
        (http://arimaa.com/arimaa/forum/cgi/YaBB.cgi)
      

        Arimaa >> Events >> 2007 Arimaa Challenge Match
        
(Message started by: RonWeasley on Jan 24^th, 2007, 8:55am)

Title: 2007 Arimaa Challenge Match
Post by RonWeasley on Jan 24^th, 2007, 8:55am

Congratulations to bot_Bomb2005CC (David Fotland) for winning the 2007 Arimaa Computer World Championship and to newcomer bot_Zombie (Evan Dorn) for placing second. bot_Bomb2005CC has earned the right to compete in the 2007 Arimaa Challenge match with bot_Zombie as the alternate.

I am pleased to announce the human defenders of the Challenge are Fritzlein (Karl Juhnke), omar (Omar Syed), and Brendan (Brendan M). The backup human defender is naveed (Naveed Siddiqui).

The public, other than the defenders named above, is invited to play the challenger bots on the arimaa game site from January 27 through February 10. The Challenge matches begin on February 11. Rules for the Arimaa Challenge are posted on the arimaa game site.

Tournament Director

Title: Re: 2007 Arimaa Challenge Match
Post by IdahoEv on Jan 24^th, 2007, 12:49pm

Quick note: the link on the gameroom is to the 2006 challenge.

2007 Arimaa Challenge (http://arimaa.com/arimaa/challenge/2007/)

Title: Re: 2007 Arimaa Challenge Match
Post by Fritzlein on Jan 24^th, 2007, 1:18pm

on 01/24/07 at 08:55:28, RonWeasley wrote:

bot_Bomb2005CC has earned the right to compete in the 2007 Arimaa Challenge match with bot_Zombie as the alternate.

If I'm not mistaken, Bomb has not yet qualified for the Challenge Match. In order to qualify, Bomb has to outperform Zombie during the two weeks both of them are available for open play against humans. In other words, Bomb and Zombie are presently on equal footing.

Title: Re: 2007 Arimaa Challenge Match
Post by omar on Jan 24^th, 2007, 11:56pm

Unlike previous years where the winner of the computer championship earned the rights to play in the challenge match, this year the two bots which finish 1st and 2nd in the computer championship will play pre-challenge games against human opponents to determine which bot plays in the challenge match.

From the challenge match page:

Quote:

The two programs which finished first and second in the 2007 World Computer Championship will compete for the right to play in the Arimaa challenge match. The two programs will be made available online for humans to play against for a duration of two weeks. Any human player that wants to play against the programs must play both programs equally in terms of number of times and color. A human player must not play more than two times against either program and no more than one time with the same color. Thus, if a human player plays program A as gold, the player must then play program B as gold. The program which has a better record after the two weeks will go on to play in the challenge match. In case of a tie the program which won the computer championship will go on to play in the challenge match. The games will be played using the 2/2/100/10/8 time control.

This thread is where we decided to use this method for selecting which bot plays in the challenge match:
http://arimaa.com/arimaa/forum/cgi/YaBB.cgi?board=talk;action=display;num=1131752654;start=0

Title: Re: 2007 Arimaa Challenge Match
Post by RonWeasley on Jan 25^th, 2007, 8:34am

Sorry I missed this intermediate qualifying step.

It seems we really want to encourage our medium-rated players to play these games so that the bots can score some occasional wins. We all may not realize that the Challenge bots are not being quarantined. If those bots often do an open-ended invite, that would help.

Title: Re: 2007 Arimaa Challenge Match
Post by Fritzlein on Jan 27^th, 2007, 3:52pm

Is the link "Please participate in the Arimaa Challenge bot qualifer" from the gameroom not working? I mean, for me it is not working, because I'm not supposed to play the bots until the Challenge, but is it not working for other people as well? I ask because some people are even playing Bomb2004CC, so tedium can't be the reason Zombie and Bomb2005CC are getting no plays so far.

Title: Re: 2007 Arimaa Challenge Match
Post by 99of9 on Jan 27^th, 2007, 8:50pm

I just tried, and I got this message:
"about to start game against bot_Zombie with side b x="

but nothing happened, and I couldn't find the game anywhere.

Title: Re: 2007 Arimaa Challenge Match
Post by 99of9 on Jan 27^th, 2007, 11:40pm

Thanks for fixing it omar, but now the problem is the two bots have different time controls (the reserve is different). Zombie appears to have the wrong control.

Ron, can I appeal to you to still count my game against Zombie? It was still 2 min per move, and I'd rather not play it all again.

Title: Re: 2007 Arimaa Challenge Match
Post by IdahoEv on Jan 28^th, 2007, 4:15am

I have confirmed that the link correctly disallows me from playing. :-)

Title: Re: 2007 Arimaa Challenge Match
Post by RonWeasley on Jan 28^th, 2007, 2:43pm

The reserve difference should be the same, but knowing how they use it, as in they really don't, I rule the games count.

Tournament Director

Title: Re: 2007 Arimaa Challenge Match
Post by 99of9 on Jan 31^st, 2007, 4:53pm

If I'm following the games correctly, so far nobody has had a split result against the bots. Everyone has either won both or lost both.

That means that to make this round decisive, we need more people to play.

Title: Re: 2007 Arimaa Challenge Match
Post by Fritzlein on Jan 31^st, 2007, 9:43pm

on 01/31/07 at 16:53:36, 99of9 wrote:

If I'm following the games correctly, so far nobody has had a split result against the bots.

Apparently just as you were writing that message, it became untrue: woh split his first two games. Bomb now has a one game lead among the four completed pairs, 3-1 to 2-2. Still, it would be nice to have as many qualifying games as possible.

Title: Re: 2007 Arimaa Challenge Match
Post by 99of9 on Jan 31^st, 2007, 10:35pm

Ok great. I also didn't realise we could play again with the opposite colours. Maybe I'll have another bash this weekend.

Title: Re: 2007 Arimaa Challenge Match
Post by Fritzlein on Feb 4^th, 2007, 10:22am

Hey, Omar, that "View Games" link is nice. There were getting to be enough pairs that I was losing track of the standings. Now it's easy to get an overview, of both complete and incomplete pairs. With nine pairs completed, Bomb leads by two games, but if clauchau beats Bomb with the bait-and-tackle after it didn't work on Zombie, then Bomb's lead will be cut to one game.

Title: Re: 2007 Arimaa Challenge Match
Post by omar on Feb 14^th, 2007, 8:02am

Link to the pre-challenge games for future reference.

http://arimaa.com/arimaa/wcc/2007/showBestBot.cgi

Title: Re: 2007 Arimaa Challenge Match
Post by Fritzlein on Feb 14^th, 2007, 1:06pm

For the record, I was afraid the qualifying phase could result in catastrophe. Bomb is clearly the stronger bot, but there seemed to be enough randomness in the process that Zombie could win "accidentally". However, with ten game pairs tied and the two decisive pairs won by Bomb, the bot that "should have" won actually did win by a comfortable margin. We'll see what happens in the future if the two bots are closer in strength.

I'm happy that eight different people played at least one pair. I expect that next year the number will be higher as folks get used to the concept, which hopefully will make the result more and more reliable.

Most importantly, however, the qualifying games fully served the purpose of not letting a new bot sneak up on us. The pairs where humans won both games, although they didn't provide discrimination, were proof of concept of various ways of winning. For example, even supposing we had had no prior experience against Zombie, Belbo's awe-inspiring 28-move immobilization would be a demonstration that the Challenge defenders could draw on. Similarly for chessandgo's 18-move win by goal. Any time there is a new bot on the scene, the "qualifying phase" will be critical as the first leg of humanity's defense.

Title: Re: 2007 Arimaa Challenge Match
Post by IdahoEv on Feb 14^th, 2007, 8:11pm

I think, frankly, there should be more time between the WCC and the challenge to allow humans -- including the defenders -- to play against the bots.

Even with this system, there is some risk of a programmer "sneaking up" on the humans, by developing a new strategy or approach that was not overall better, but was merely surprising.

Imagine some developer manages this in the 2010 challenge, and wins the prize. And then within a week of study afterwards, the defenders had reliably figured out how to beat the bot. Did the bot really deserve the prize that year? It seems to be the prize should be given to the first bot that can consistently beat top players, not just one that sneaks up.

Yes, this means that the humans have an opportunity to learn from their early losses to the bot. But, it also means the bot has an opportunity to learn from its losses as well. And, it seems to me, any bot likely to truly win at this game is going to need to do just that anyway.

I would put the challenge something like two full months after the CC, and let everyone including the defenders play against the bots.

Title: Re: 2007 Arimaa Challenge Match
Post by IdahoEv on Feb 14^th, 2007, 8:19pm

on 02/14/07 at 13:06:08, Fritzlein wrote:

For the record, I was afraid the qualifying phase could result in catastrophe.

Define "catastrophe": Zombie wouldn't have beaten the defenders either.

But I see your point. While at this point it is fairly meaningless which bot plays against the humans, there were an awful lot of timeout games going into the "score" on either side. Unfortunately, I don't see an answer to that.

Title: Re: 2007 Arimaa Challenge Match
Post by Fritzlein on Feb 14^th, 2007, 9:22pm

on 02/14/07 at 20:19:00, IdahoEv wrote:

Define "catastrophe": Zombie wouldn't have beaten the defenders either.

By catastrophe I meant that the clearly weaker bot would win by luck. The closer the two bots are in strength, the less it matters which one wins the qualifying, and in fact it is reasonable that a bot which is slightly weaker head-to-head would be slightly stronger against humans.

Title: Re: 2007 Arimaa Challenge Match
Post by Fritzlein on Feb 14^th, 2007, 10:56pm

on 02/14/07 at 20:11:56, IdahoEv wrote:

Imagine some developer manages this in the 2010 challenge, and wins the prize. And then within a week of study afterwards, the defenders had reliably figured out how to beat the bot. Did the bot really deserve the prize that year? It seems to be the prize should be given to the first bot that can consistently beat top players, not just one that sneaks up.

I agree that the Arimaa Challenge could be won by software that was not absolutely dominant over humans.

There are lots of ways to measure dominance. With ever-increasing hardware speeds, it is tempting to settle on a very high standard, such as being able to beat all human players with greater than 50% probability, not only today, but for all time, without further developer intervention to fix bugs and add new strategic knowledge that humans may pick up in the mean time.

I certainly don't think we need such a high standard to say that computers have passed up humanity at playing Arimaa, but let's say we really do want the Challenge to measure this standard. My hunch is that software which can beat three top humans each two games out of three (after having been exposed to community play for two weeks) must be very close, even if it isn't all the way there. If humans soon discover a reliable way to beat such software, it is probably due to a minor quirk that can soon be fixed rather than seriously flawed strategy which worked only due to surprise. Our understanding of Arimaa is probably mature enough that there isn't some strategy trick nobody has thought of yet, which pushes a bot past us only temporarily without heralding the end of human dominance. In all likelihood, the error of the current format would only be in awarding the prize one year too soon.

But this isn't the main reason I oppose making the Challenge tougher to win. My main issue would be that it isn't fair to keep moving the goal posts. When Omar first put out the Challenge the bot was only required to win a match against one top human. In the third year he changed it to require the bot to beat each of three humans. This moved back the goalposts significantly from the original requirement.

Suppose there really were a dominant bot that could beat all humans 60% of the time (and obviously win even more often against lower-ranked humans). The chance that it would win an eight-game match against one top human would be 59.4%, even with 4-4 ties going to the human. But this same dominant computer would only have a 27.2% chance of winning all three mini-matches against three top humans. Probably one human would get lucky and win twice even if the computer had a 60% chance to win each individual game.

In other words, Omar has already moved the goal posts once, and made it so that a bot that is clearly the best player in the world could have a good chance of not winning the Challenge prize. I believe Omar when he says that the current goal line is closer to what he had in mind all along, but it introduces a significant chance that the Challenge prize will be awarded too late, an error even by the stringent standard of dominance first outlined.

Furthermore, the fact that there is a qualifying round at all has toughened the requirement on bots. In 2005 developers only had to have their bots play at least six games total against at least three different humans, and the strength of the humans was not specified. For a bot to be open to play for two weeks against all humans (up to four games each) allows much more exposure, and much more probing for weaknesses. (To be honest, the developers probably don't mind so much since it takes the responsibility of finding human opponents off their shoulders, but for a developer playing purely to win, winning got harder.)

If we now extend the qualifying period to two months instead of two weeks and also allow the actual defenders to play as many games as they like against the bots, then it will just appear to be part of the general trend of making the Challenge rules more difficult every year.

Yes, there is always a case to be made that, until a program can do X, it isn't totally dominant. Chess developers suffered for years from man vs. machine competitions taking place under rules more and more favorable to humans. Developers could be excused for thinking that no matter how good their programs got, humans would find a way to change the rules once again and say that programs aren't really champs.

I feel this is exactly why we have to guard against making the Challenge harder and harder to meet. Omar should pick a reasonable middle ground that tries both to disallow sneak attacks and to allow a program that really is the best to claim the prize. Yes, we are still working out the kinks and finalizing the rules, but as we work out the details, the difficulty should stay basically constant and not increase every year.

Title: Re: 2007 Arimaa Challenge Match
Post by PMertens on Feb 15^th, 2007, 3:05pm

If a bot should be able to beat the defenders only to be shown to have a weakness two months later then it certainly still should get the prize imho.
It is very likely that in a fixed performance bot somebody will eventually find a weakness ... but I do not think the point of the exercise is to create a flawless bot, just one that proves that humans will eventually be outdated no matter how much they try to cheat ;-)

At the moment the bots are so incredible far away from being even close to a threat to the (top)humans, that anything capable of beating 50% would be an absolute breakthrough.
No need to move goalposts since once that happens we will all know that it is only a matter of (short) time until we lost this game to the comps.

Title: Re: 2007 Arimaa Challenge Match
Post by jdb on Feb 15^th, 2007, 6:00pm

Quote:

...Our understanding of Arimaa is probably mature enough that there isn't some strategy trick nobody has thought of yet, ...

Title: Re: 2007 Arimaa Challenge Match
Post by Fritzlein on Feb 15^th, 2007, 8:06pm

Heh, JDB, you only quoted half my sentence. I actually expect there are tons of Arimaa strategies we haven't thought of yet. If a bot surpasses humans, it will probably be by better execution of simple strategies than by bringing new strategies to the table, in my humble opinion. However, IF a bot should chance to win by bringing a new strategy to the table, THEN it will mark the end of human dominance no matter how we try to adapt. I'm not saying that we can't be surprised by anything new, just that if we are beaten by X, whatever X is, it is probably too late for X to be a cheap trick that works only one year, after which humans regain ascendancy.

Title: Re: 2007 Arimaa Challenge Match
Post by jdb on Feb 16^th, 2007, 2:03pm

Hi Fritz,

I agree with your statement. Your quote was just too juicy for me to ignore, even if it was taken out of context. :)

Title: Re: 2007 Arimaa Challenge Match
Post by Belteshazzar on Feb 2^nd, 2019, 4:47pm

on 02/14/07 at 22:56:58, Fritzlein wrote:

I certainly don't think we need such a high standard to say that computers have passed up humanity at playing Arimaa, but let's say we really do want the Challenge to measure this standard. My hunch is that software which can beat three top humans each two games out of three (after having been exposed to community play for two weeks) must be very close, even if it isn't all the way there. If humans soon discover a reliable way to beat such software, it is probably due to a minor quirk that can soon be fixed rather than seriously flawed strategy which worked only due to surprise. Our understanding of Arimaa is probably mature enough that there isn't some strategy trick nobody has thought of yet, which pushes a bot past us only temporarily without heralding the end of human dominance. In all likelihood, the error of the current format would only be in awarding the prize one year too soon.

Looks like this was largely correct.