Arimaa Forum - Print Page


    
      
        Arimaa Forum
        (http://arimaa.com/arimaa/forum/cgi/YaBB.cgi)
      

        Arimaa >> Events >> Human player for the 2005 Arimaa challenge match
        
(Message started by: mouse on Jan 14^th, 2005, 5:08am)

Title: Human player for the 2005 Arimaa challenge match
Post by mouse on Jan 14^th, 2005, 5:08am

When will the human player for the Arimaa challenge match be selected and by which criteria will the selection be done? Is this stated anywhere?

Title: Re: Human player for the 2005 Arimaa challenge mat
Post by 99of9 on Jan 14^th, 2005, 8:01am

It is supposed to be announced before the winner of the bot tournament is known (in fact just before the bot tournament). So I'd say you'll get a fairly quick answer.

The selection was entirely up to Omar and Aamir.

Title: Re: Human player for the 2005 Arimaa challenge mat
Post by mouse on Jan 14^th, 2005, 8:06am

Ok it's the only thing that make sense to make the selection before the winner of the bot tournament is found. To prevent selection issues. Not that it will make much of a difference this year since there is a handful of players who can win both against bomb and clueless.

Title: Re: Human player for the 2005 Arimaa challenge mat
Post by Fritzlein on Jan 14^th, 2005, 11:48am

The selection of the human representative is made before the computer tournament begins, and the name is submitted to the director of the challenge match. I am not sure why this name is not immediately publicly announced, because at present you just have to trust Omar and the tournament director that they didn't change their minds. But anyway, the selection was already made.

As to the criteria for choosing which human will play, it is entirely and officially up to the whim of Omar and Aamir. I suspect that at a time control of two minutes per move, there are quite a few human players who would win the challenge despite being lower-rated than Bomb. PMertens, for example, has had reasonable success against Speedy, and if he got to slow down, I'd bet on him over Bomb in an eight-game match.

Title: Re: Human player for the 2005 Arimaa challenge mat
Post by PMertens on Jan 16^th, 2005, 2:18pm

ah I feel honoured ::)

but in a 2 minute game my concentration would totally deteriorate ... so better choose somebody else ;-)

Title: Re: Human player for the 2005 Arimaa challenge mat
Post by omar on Jan 22^nd, 2005, 10:34am

The specs for the selection process can be found here:
http://arimaa.com/arimaa/challenge/2005/

I submit the name of the primary human player along with two backups to the ICGA on by Jan 4th; which is also the deadline for the developers to submit the bots. The submission is made in writting and the ICGA has a record of it. However, the announcement of who the human players are is not done until after the bot tournament is over. The ICGA posts it on their web site:
http://www.cs.unimaas.nl/icga/news/events/Arimaa/arimaa.html

But this year Jeroen seems to have forgotten; I've sent an email to remind him.

Anyways this years primary is Frank (Belbo); first backup is Karl (Fritzlein) and I am the second backup. The backups are there in case the primary player gets sick or has network or computer problems and is not able to play.

Since I already played in the challenge match last year, I wanted someone else to play it this year. I checked with Karl, Toby and Frank. Both Karl and Toby said they would like to do it, but could not make the time commitment to play eight games; they could probably play a few games but not the whole series. Frank was a bit concerned about taking on such a big responsibilty, but I convinced him that I was sure he could do it. Last year also I had asked Frank to be the human rep, but at that time also he did not feel comfortable taking on the responsibility.

The reason for not announcing the human opponent before the bots are submitted is just so that the bot is not tuned to a play a particular human. However the bots are provided the name of the opponent as part of the gamestate data so they could be tuned against several possible human candidates. Also since the name of bots always begins with bot_ a bot can use this to play differently against humans than it does against other bots; I don't mind a bot doing that. But in general I want to discourage tuning a bot to play against a particular human opponent just to win the challenge prize. The challenge prize should be won when a bot gets to the point where it can defeat any human opponent regardless of who it is.

When I wrote the specs for the format of the challenge match a couple years ago I didn't have any practical experience with the selection process. After having gained some experience now I am finding that it is going to be hard to find a human opponent for the challenge match; either due to time constraints or not wanting to take on the burden of defending the challenge. Thus I have been think lately about changing the format for the challenge match.

Here is what I am currently considering:
I will select a team of about 3 to 5 human opponents that will each play a series of 3 games against the computer opponent. To win the challenge the computer must defeat all the humans (i.e. win two of the three games against each human opponent). The game schedual will be such that all the human opponents play the first game before any of them starts the second game and likewise for the third game. If the computer has already been defeated then the remaining games become optional and don't need to be played.

I think this format will eliminate the difficulty of finding human opponents for the challenge match.

Title: Re: Human player for the 2005 Arimaa challenge mat
Post by Fritzlein on Jan 22^nd, 2005, 12:24pm

It would be an interesting format to have the computer challenger trying to win 2 of 3 from multiple human defenders. But it seems odd to make it enormously more difficult to win the challenge in the process. Isn't the issue to make it easier to find humans to play, rather than to make it vastly harder for the program to win?

Let's say a program gets to where it can win 50% of the time against top humans. Then it would have a 50% chance of winning the challenge match in the current format. But if it had to win best of three against all of four different opponents, it would have only a 6.25% chance of winning the challenge.

I think a fairer way to alter the challenge would be to have three human defenders and for the program to win it would have to win two of three games from two of the three humans. That way a program which is able to win 50% of its games will still have a 50% chance of winning the challenge. It stays fair.

One can also roughly calculate whether the old eight-game format would be more or less conducive to upsets than my proposed three-by-three-game format. For example, suppose the program wins only 40% of its games against top players. Then in the 8-game format the program would have a 17% chance of winning and 23% chance of tying, for about a 29% chance of an upset overall, assuming it wins about half the ties. In the 3-by-3 format the program would have no chance of tying and a 28% chance of winning an upset overall.

So apparently the 3-by-3 game format would keep the difficulty of winning the challenge approximately what it is now, but would make it easier for players to commit to playing. One might argue that it makes an upset more likely if you have to involve the top three humans rather than just letting the top human defend, which is true, but it is less of an effect than if the top human is unable to commit to a full eight games and the whole weight falls on the next guy, as it has this year.

One other thing you might do, Omar, is give yourself the choice of having a single human defend in an eight-game match OR having three humans play three games each. Maybe some years one will be easier to organize and some years the other will be. As long as you decide before the bot tournament which it will be, it should be fair to the bots, because the win probability and upset probability of each is about the same.

Of course, since it is your challenge you can do whatever you like, but it would seem odd to massively raise the bar when in fact all you want to do is change the format.

Title: Re: Human player for the 2005 Arimaa challenge mat
Post by 99of9 on Jan 24^th, 2005, 7:57am

Go Belbo! We know you'll represent us well. (and Fritz and Omar if you end up involved).

I agree with Fritz that Omar's proposal is a gigantic raising of the bar for the bots, just as bots were (slowly) approaching the original bar. As such I don't think it is good.

Nevertheless, I appreciate the drive to put less time and energy pressure on the primary human - especially since I'm one of the ones who defaulted this year.

I would propose preserving the 8 game match, but allowing Omar the flexibility to choose (in advance?) which humans will play which games. It might be 1 player, it might be 8. Players can consult with Omar on their February schedules, and then he's more likely to get the top players to be able to play a few games.

I like this because:

it doesn't alter the probabilities
it keeps the original structure, so can't be conceived of as cheating
it preserves the gold-silver balance (important in my view)
it preserves the slight human advantage that a 4-4 tie does not win the challenge
it allows lots of flexibility on the human side
if the humans do end up losing one year, no single person feels the overwhelming burden of guilt at costing Omar $10,000

The only thing I'd be a bit dubious about is allowing one player to play all gold games, and another to play all silver games.

Sorry Fritz I don't like your suggestion that much, because it breaks things down into mini-challenges... I like the idea of one grand match. Even if more than one human is playing it, it seems aesthetically grander to keep all the scores in the same pot.

Title: Re: Human player for the 2005 Arimaa challenge mat
Post by Paul on Jan 25^th, 2005, 5:05am

I really like Toby's idea. It's too an easy way to have an important part of the players involved in the challenge each year.

I think we can even go a little farther : each player that has a rating greater than the winning bot should be allowed to play one game of the challenge.

The challenge makes a lot more sense this way for me, as there is less psychological consideration for the humans (for exemple, the pressure on a single human player will be huge if he lost the two first games...)

Title: Re: Human player for the 2005 Arimaa challenge mat
Post by omar on Jan 28^th, 2005, 6:52pm

Im glad you guys brought up the issue of the difficulty of the challenge. I probably should have discussed it in my earlier post, but it was getting late and the post was getting too long as it is.

Unforturnately I won't be able to discuss it now either as I am running late. But I do have to say something more about it. For now I will just mention that my original intent for the Arimaa challenge is that:

The challenge prize should be won when a bot gets to the point where it can clearly defeat any human opponent regardless of who it is or how strong they are.

Title: Re: Human player for the 2005 Arimaa challenge mat
Post by RonWeasley on Jan 31^st, 2005, 11:13am

"The challenge prize should be won when a bot gets to the point where it can clearly defeat any human opponent regardless of who it is or how strong they are."

This was the intent I remember when I first learned about arimaa in Games Magazine. The implicit assumption is that some technology breakthrough will be required for the bot to succeed. I would even go as far as to allow an "open season" on any bot that wins the challenge match. During this period, maybe a month, any player can try to defeat it. The prize would be won if the bot remains undefeated.

This is more difficult than the original criteria, but truer to the intent and it makes it more likely Omar gets his $10,000 worth of breakthroughs.

Title: Re: Human player for the 2005 Arimaa challenge mat
Post by omar on Feb 2^nd, 2005, 7:47am

The Arimaa challenge as it currently stands is not as difficult as it should be and the new format helps to make it more difficult. I didn't discuss this aspect of the new format in my original post since it is a topic in itself.

The Arimaa challenge match is really just trying to implement the intention I mentioned above. That is the challenge prize should be won when a bot gets to the point where it can clearly defeat any human opponent regardless of who it is or how strong they are.

If I had it my way I would have every person in the world learn to play Arimaa and play a series against the bot to make sure that the bot could really defeat any and all humans, before it won the challenge prize :-)

Of course that is not practical, but consider releasing a program with the claim that it can defeat anyone in a series of say 5 games played at the 2 min per move time control. Imagine that anyone who wants to can take one crack at playing it best of 5. Similar to what Ron just proposed. If someone could show that they won the series then the program would not win the challenge prize. This is closer to being doable, but lacks being clean and official.

The grandest way of doing it is to determine who the very best human player is and have that person play the series to defend the challenge. This is much more practical. But the best Arimaa player may not be available to play, might not want to play, or might even be the same person who wrote the best program (and have a conflict of interest). Thus, the need for someone to be selected to represent the humans.

Although this third format seems in principle to be equivalent to the first it is actually much weaker. The proposed format is a small step closer to having the program play all humans and helps significantly to make the challenge stronger.

There are several reasons why the current Arimaa challenge is so weak. First of all the most talented players in the world have probably not even discovered Arimaa yet. The population of Arimaa players is so low that the probability that the most talented player is actively playing Arimaa is less than one in a million. Secondly among the players that are actively playing Arimaa the probability that the best one will be selected to play in the challenge match is also fairly low. Finally it takes human many years to master complex strategy games. Even with hundreds of years of knowledge about the game and master level teachers available it takes even talented players many years of active playing and studying to reach a significantly high level of play. So even if the most talented players were actively playing Arimaa today, it would still take many years for them to achieve what would be considered a very high level of play in the game. Keep in mind that the current players are just playing for fun and not for a living.

Thus even if Arimaa was a more difficult game for computers than Go, the task of developing a program to defeat the best Arimaa player is significantly easier than the task of developing a program to defeat the best Go player. Even with the new proposed format for the challenge match I believe it still remains easier.

continued ...

Title: Re: Human player for the 2005 Arimaa challenge mat
Post by omar on Feb 2^nd, 2005, 7:49am

(got an error that my message was too long and to shorten it :-) )

When I originally released the Arimaa challenge I had not considered these factors and realized it later. Over the last couple of years I've known that the Arimaa challenge needs to be made more difficult to compensate for the "newness" of the game, but I didn't know how to do it. Then when I was looking for ways to solve the problem of finding players for the challenge match I realized the new format would also make the challenge more difficult. It solved both problems at once.

Now one thing that I do need to be careful about is not making the Arimaa challenge impossible to achieve within the scope of it's time frame. The hope is that if the Arimaa challenge is achieved, it will be done by significant advances made in software and not just due to the advances in hardware; thus the time limit of year 2020. When the challenge is made more difficult it means that the performance of the program must be significantly more than the performance of the human players to achieve it. Indirectly it raises the question of what is the level of perfect play and how close the humans will be to that level within the time frame of the challenge. Will there be enough room between the best human players and the limit of perfect play for a program to demonstrate that it is clearly better? I think that within the time frame of the challenge, Arimaa will still be such a new and unexplored game that there will be sufficient room for a program to demonstrate that it has achieved a significantly higher level of play than the humans.

Now the downside of making the challenge more difficult is that there might be a lack of interest in it. In fact I'm pretty sure that David Fotland took on the challenge, because he was wise enough to see that the Arimaa challenge could be easy to achieve just due to the newness of the game.

To compensate for the challenge being more difficult I plan to solicit sponsors to help increase the challenge prize. My original plan was that I would not be the only one contributing to the Arimaa challenge prize. I had hoped that others, including individuals and corporate sponsors would also make a pledge to increase the challenge prize. However, I did not pursue this when I realized that the challenge was not too difficult to achieve. Now that the challenge has been tested and with the new stronger format I would feel more confident to ask others to make a pledge to help increase it. I would love to see the challenge prize be around a million dollars someday :-)

Omar

Title: Re: Human player for the 2005 Arimaa challenge mat
Post by fotland on Feb 4^th, 2005, 12:44am

I did work hard the first year because I felt that the challenge would be easiest to win in the first year, when human players were still weak. I thought it would be enough to have a program that was tactically strong, at traps and goals, since the top humans would still blunder. However, Omar got too strong during the first year, and bomb lost all its games. This year, I focused on strategy, but it is still not enough to win.

I think you will get a lot of interest if you increase the yearly top program prize. The Ing competion had a $1 million prize, but everyone knew it would never be won. Still, there were many programs written because the yearly prize for the top program was about $5,000.

If you want AI breakthroughs, I think you will need to interest the Universities. Ph.D's are willing to explore. People like me, who are trying to win through superior engineering, will tend to stick to provent techniques.

There will be plenty of room between top humans and perfect play. Arimaa is a deep enough game that people will be far from perfect.

Title: Re: Human player for the 2005 Arimaa challenge mat
Post by Fritzlein on Feb 4^th, 2005, 3:50pm

I second the notion that we don't have to worry about humans getting too close to perfect play any time soon.

I suppose that the game still could be "busted" by some simple, powerful ideas that take out all the strategy and complexity. The closest thing to that so far was when it became unpopular to attack an opposing trap and it looked like Arimaa might boil down to a defensive draw. Perhaps that will still happen if we collectively decide that the popular attacks of today are all unsound or preventable, but I doubt it. It appears that even if the (unpreventable) elephant/horse attack is unsound, defensive play can't protect all the rabbits, because either the back central rabbits or the advanced flank rabbits will be vulnerable.

Many factors seem to indicate we are not yet anywhere near perfection. In terms of the game itself we can note:
* There has been only 1 draw in over 10000 games
* The decisive games have been within 1% of evenly split between Gold and Silver
* Even the most equal of games seem to become unbalanced (in ways mysterious to us) as they progress rather than stabilizing.
* The games don't all follow the same pattern, so it appears we aren't narrowly constrained in the choice of good moves.

And in terms of our progess we also see:
* Top-rated players routinely make strategic blunders
* Top-rated players sometimes get beaten without anyone being able to pinpoint a blunder