Author |
Topic: Arimaa Challenge format for 2006 (Read 4772 times) |
|
99of9
Forum Guru
Gnobby's creator (player #314)
 
Gender: 
Posts: 1413
|
 |
Re: Arimaa Challenge format for 2006
« Reply #1 on: Aug 15th, 2005, 6:43pm » |
 Quote  Modify
|
This post is about to get extremely harsh. I just want to say upfront that I admire the effort and generosity that you put into this game Omar, and I do not doubt your integrity, or motives for making this change. But I still want to say these things, because I think going ahead with this would be a blunder:
I still think that it is a bad idea to change the challenge to make it much harder. Even if that was your original intent (although I for one did not know that it was until this year). Moving goalposts just don't instill confidence or trust that they will not move again every year. Your intent is so all-encompassing that it could be used to justify almost any increase in difficulty. Here is the intent you stated on the other thread, just so we don't have to refer back:
Quote:| The challenge prize should be won when a bot gets to the point where it can clearly defeat any human opponent regardless of who it is or how strong they are. |
|
What if I were to propose a format where every human on the arimaa list played one game (with gold) against the bot, and if the bot lost a single game, it lost the challenge. I think that would be supremely unfair, but it could be justified as more likely to satisfy your intent.
In fact even modern computer-chess programs would probably not win your proposed challenge format. To me, humans have clearly been surpassed in chess (not all areas of it, but as a whole), but they are still capable of winning one out of three 3-game matches.
I thought we were claiming that arimaa was orders of magnitude more difficult for computers than chess? If that is really the case, then any defeat of humanity (in the old challenge format) would indicate that either we are wrong, or that AI has improved.
If the challenge keeps getting harder, can we really say that AI is not making advances in leaps and bounds? No.
One of the problems with getting AI accepted in the general community is that each time an intelligence problem is proposed, people say "no way a computer could do that - that requires true intelligence". Then some clever computer scientist achieves it, and the people say "that was just computation that you did, brute force, therefore that's not intelligent". So AI is forever on the horizon, and all the real advances are not credited as such, because the goals have moved.
For what it's worth, I think some of the stuff that Jeff and Haizhi are doing is actually an AI advance, although admittedly not in as grand a way as you were initially anticipating.
Quote:| Omar and Aamir Syed reserve the rights to modify the challenge details in the future to resolve any logistical problems that may arise. |
|
I'd say that this modification is not purely a logistical change. Thus if you go ahead with this, the possibility of challenge-difficulty-altering changes should also be mentioned in the rules disclaimer.
Quote:To compensate for the challenge being more difficult I plan to solicit sponsors to help increase the challenge prize. My original plan was that I would not be the only one contributing to the Arimaa challenge prize. I had hoped that others, including individuals and corporate sponsors would also make a pledge to increase the challenge prize. However, I did not pursue this when I realized that the challenge was not too difficult to achieve. Now that the challenge has been tested and with the new stronger format I would feel more confident to ask others to make a pledge to help increase it. I would love to see the challenge prize be around a million dollars someday  |
|
Just for interest, was there any progress on this?
|
| « Last Edit: Aug 15th, 2005, 8:25pm by 99of9 » |
 IP Logged |
|
|
|
jdb
Forum Guru
Arimaa player #214
Gender:
Posts: 682
|
|
Re: Arimaa Challenge format for 2006
« Reply #2 on: Aug 15th, 2005, 7:49pm » |
Quote Modify
|
In my opinion, winning two out of three games against all three selected opponents seems fair enough to me.
|
|
IP Logged |
|
|
|
Fritzlein
Forum Guru
Arimaa player #706
Gender:
Posts: 5928
|
|
Re: Arimaa Challenge format for 2006
« Reply #3 on: Aug 15th, 2005, 9:28pm » |
Quote Modify
|
I'm afraid I agree with 99of9 in thinking that the new challenge rules will be very detrimental.
What catches the attention of the public is the basic idea, "Hey, there's a boardgame where humans can still beat the computers!" That's more important than the dollar amount of the prize, or the length of the match, or the structure of the match. The idea is that Arimaa is a game of real intelligence. That's what the Web site says, and that's the only reason most people pay any attention to Arimaa at all.
Suppose that you offered a prize of one million dollars that no bot could beat the top human 10 games out of 10 in a match. That huge amount of money would be less impressive than the ten thousand dollar prize has been, because there is no emotional impact to saying "Computers are not totally infallible at Arimaa," or "Human intelligence is occasionally useful in Arimaa." Who cares? The gut-check question is whether man or machine is better.
I can foresee the following disaster scenario with the new rules: Some year the bot will score 3-0, 1-2, and 3-0 against the human defenders respectively, for a total score which is a dominant 7-2 victory over the humans. Now, because of the way the match is structured, the prize money won't be awarded, but so what? Nobody will care. It will be all over in the minds of the public. There will be no headlines like "Humans regain dominance in Man vs. Machine" or "Humans can still outsmart computers". It's not interesting to anybody that "Humans are not totally humiliated at Arimaa yet." If that happens Arimaa will instantly lose all of its cachet anyway, so you might as well give the prize to the bot developer then.
Admittedly, the old rules which pit the best computer against a single human didn't 100% precisely answer the question of whether Arimaa is a game of real intelligence, but that's what that match stood for to the masses. If you want to keep that flame of interest alive, I think the new structure has to give out the prize money as soon as computers are "better than humans". If there are three mini-matches against top humans, and if the computer wins two out of three of those mini-matches, then machine has already triumphed over man no matter what the rules say.
Like 99of9, I must apologize for sounding harsh about your plans, Omar. Of course, since it is your game, your server, and your money, you can do whatever you like. You are legally and morally in position to have the Arimaa Challenge be what you want it to be. My point is only that if you change it the way you are proposing, it will take the air out of Arimaa's sails. It will detract from Arimaa's aura of being something special that humans are good at.
|
| « Last Edit: Aug 15th, 2005, 10:13pm by Fritzlein » |
IP Logged |
|
|
|
PMertens
Forum Guru
Arimaa player #692
Gender:
Posts: 437
|
|
Re: Arimaa Challenge format for 2006
« Reply #4 on: Aug 15th, 2005, 9:48pm » |
Quote Modify
|
I absolutely agree with the former posts.
In addition I would like to mention that there is absolutely no need to raise the bar for the bots.
The current champion can not only be outplayed in one but in several different ways with all kinds of handicaps.
Currently David would probably not even need to change anything to win again, but even if he does it is highly doubtful that we wont find an exploit before the challenge match.
Depending on who will play this year you might even see a handicap game during the challenge.
|
|
IP Logged |
|
|
|
Fritzlein
Forum Guru
Arimaa player #706
Gender:
Posts: 5928
|
|
Re: Arimaa Challenge format for 2006
« Reply #5 on: Aug 15th, 2005, 10:38pm » |
Quote Modify
|
on Aug 15th, 2005, 4:37pm, omar wrote:Another thing which I changed is the selection of the human players. I decided to look at the record of the players against the previous years bots to determine who the selected players will be. This makes the selection process less subjective.
|
|
I like the new formula! This thing I like best about it is that we will now have a sort of bot-bashing contest between humans to see which of us will qualify to defend the honor of humanity. The people who win that contest will be people who really want to smash the championship bot, as opposed to people who have to be reluctantly pressed into service.
I like that you key on the minimum number of wins across all four bots. It makes sense that you should be good against all bots to make the human team.
I do have a few minor suggestions, though. (Don't I always? ) I think that the effectiveness should not be calculated by 100*W/(W+L), as normal percentages are. A 50% score should be neutral, and anything below 50% should count against you. Therefore instead of percentage, we should use 100*(W-L)/(W+L). Right now my score of 3.25 could be beaten by someone who scores 7 of 15 against each bot. But having a losing record against the bots primarily shows that you are not fit to defend the challenge, even though you are practicing a lot.
My second suggestion is that for the four championship bots from last year, since they are being used as a qualifying benchmark, unrated mode should be suspended. Every game against those bots should be rated. In particular, it is possible to get a favorable initial piece setup in botbashing by repeatedly resigning games on the second move until the bot randomly places the pieces where you want them. Those early resignations, which currently abort the game with no record, should be forbidden (i.e. they should count as losses so no one will do it). The type of botbashing to qualify for the human team should be about consistent winning, not about the spectacular death-defying feat you can only get one time out of ten.
And finally, I suggest that you wipe the slate clean and start everyone from scratch. I think that would be fairer to people who might want to qualify, but didn't know what the rules were going to be until now.
Anyway, whatever the rules are for qualifying, this year I'm going to try hard to make the human team. I want to be part of the squad that kicks some bot behind.
|
|
IP Logged |
|
|
|
99of9
Forum Guru
Gnobby's creator (player #314)
Gender:
Posts: 1413
|
|
Re: Arimaa Challenge format for 2006
« Reply #6 on: Aug 16th, 2005, 6:54pm » |
Quote Modify
|
on Aug 15th, 2005, 10:38pm, Fritzlein wrote:| My second suggestion is that for the four championship bots from last year, since they are being used as a qualifying benchmark, unrated mode should be suspended. Every game against those bots should be rated. In particular, it is possible to get a favorable initial piece setup in botbashing by repeatedly resigning games on the second move until the bot randomly places the pieces where you want them. Those early resignations, which currently abort the game with no record, should be forbidden (i.e. they should count as losses so no one will do it). The type of botbashing to qualify for the human team should be about consistent winning, not about the spectacular death-defying feat you can only get one time out of ten. |
|
I agree that aborted rated games should probably count (although it's probably safe to trust humans with that for the moment - I don't think the competition will be so intense that people resort to this kind of cheating.
I'm not sure I agree with counting unrated games, because then no one will play handicap botbashes against those bots until a year after they are designed.
|
|
IP Logged |
|
|
|
Fritzlein
Forum Guru
Arimaa player #706
Gender:
Posts: 5928
|
|
Re: Arimaa Challenge format for 2006
« Reply #7 on: Aug 16th, 2005, 8:44pm » |
Quote Modify
|
on Aug 16th, 2005, 6:54pm, 99of9 wrote:| I'm not sure I agree with counting unrated games, because then no one will play handicap botbashes against those bots until a year after they are designed. |
|
Oh, well, now that I think about it, allowing unrated games isn't so bad. If someone wants to practice a few times to improve technique before risking a rated game, why not? At least it would show that they were serious about the challenge.
|
|
IP Logged |
|
|
|
PMertens
Forum Guru
Arimaa player #692
Gender:
Posts: 437
|
|
Re: Arimaa Challenge format for 2006
« Reply #8 on: Aug 16th, 2005, 10:31pm » |
Quote Modify
|
on Aug 15th, 2005, 10:38pm, Fritzlein wrote:
Anyway, whatever the rules are for qualifying, this year I'm going to try hard to make the human team. I want to be part of the squad that kicks some bot behind. |
|
Well, you are currently leading the team ... and only 2 other persons qualify (but 2 more trying to get in ?)
How about creating an official competition with a direct link from the main page ?
|
|
IP Logged |
|
|
|
omar
Forum Guru
Arimaa player #2
Gender:
Posts: 1003
|
|
Re: Arimaa Challenge format for 2006
« Reply #9 on: Aug 17th, 2005, 2:56am » |
Quote Modify
|
Trying to find a human player for last years challenge match turned out to be more of a challenge than I had expected. I realize now that asking one person to play 8 intense games with the preasure of defending the challenge prize is a bit much. So the obvious solution was to have multiple players to distribute the games among. But once you have multiple humans player with each playing an equal number of games against the bot a new issue arises as to what the criteria should be for the bot to demonstrate that it is better than even the best human. The three obvious ones are:
1. The bot wins if it wins the majority of the games
2. The bot wins if it wins the individual series against the majority of the human players
3. The bot wins if it wins individual series against each of the human players
The first two options do not change the difficulty of the challenge match, but the third option does. The more human opponents there are the more difficult it becomes. However, it is the only option that seems logical in demonstrating that the bot is clearly better than any and all humans. Karl pointed out in the other thread that if there were 4 human players then the chances of the bot winning would decrease to 6.25% if we use the third option.
I mentioned in the thread that the Arimaa challenge as it currently stands is not as difficult as it should be and the new format helps to make it more difficult. I will restate the reasons again here in case anyone has not read that thread.
There are several reasons why the current Arimaa challenge is so weak. First of all the most talented players in the world have probably not even discovered Arimaa yet. The population of Arimaa players is so low that the probability that the most talented player is actively playing Arimaa is less than one in a million. Secondly among the players that are actively playing Arimaa the probability that the best one will be selected to play in the challenge match is also fairly low. Finally it takes human many years to master complex strategy games. Even with hundreds of years of knowledge about the game and master level teachers available it takes even talented players many years of active playing and studying to reach a significantly high level of play. So even if the most talented players were actively playing Arimaa today, it would still take many years for them to achieve what would be considered a very high level of play in the game. Keep in mind that the current players are just playing for fun and not for a living.
Thus even if Arimaa was a more difficult game for computers than Go, the task of developing a program to defeat the best Arimaa player is significantly easier than the task of developing a program to defeat the best Go player. Even with the new proposed format for the challenge match I believe it still remains easier.
I also mentioned that I do need to be careful that the new format does not make the Arimaa challenge impossible to achieve within the scope of it's time frame. The hope is that if the Arimaa challenge is achieved, it will be done by significant advances made in software and not just due to the advances in hardware; thus the time limit of year 2020. When the challenge is made more difficult it means that the performance of the program must be significantly more than the performance of the human players to achieve it. Indirectly it raises the question of what is the level of perfect play and how close the humans will be to that level within the time frame of the challenge. Will there be enough room between the best human players and the limit of perfect play for a program to demonstrate that it is clearly better? I think that within the time frame of the challenge, Arimaa will still be such a new and unexplored game that there will be sufficient room for a program to demonstrate that it has achieved a significantly higher level of play than the humans.
There might be concern that the challenge rules may be changed again in the future to make the challenge more difficult. Although I was careful to leave that option open in the early years by adding the disclaimer, I want to remove that option completely and the challenge rules should be practically etched in stone. I would like to remove that option next year if things go smoothly this year and there is no problem in finding players.
|
|
IP Logged |
|
|
|
Fritzlein
Forum Guru
Arimaa player #706
Gender:
Posts: 5928
|
|
Re: Arimaa Challenge format for 2006
« Reply #10 on: Aug 17th, 2005, 11:17am » |
Quote Modify
|
I understand the resons for your decision Omar. I don't think the general public will perceive it in same way you do, but that's OK. With luck, the humans will score a 9-0 shutout victory this year to keep alive among the masses the idea that Arimaa is hard for computers to play well. Furthermore, if the gap between humans and computers continues to widen (as it seems to be doing at the present) there will be more shutouts in the years to come. If that happens, the exact criteria for winning the challenge become less important.
on Aug 17th, 2005, 2:56am, omar wrote:[...] if things go smoothly this year and there is no problem in finding players.
|
|
I don't know how other people feel, but the new rules make a huge difference in my eagerness to play. It is hard to play four games per week for two weeks in a row if one has a full-time job. But now, if the challenge is spread across three weeks, each human only has to play one game per week. (The bot presumably doesn't get tired, so it can play three times per week.) Playing one official game per week should be no problem at all, even if work is busy.
This year you won't have to invite me to play: I'll be actively trying to qualify. I want to be part of the 9-0 demolition!
|
|
IP Logged |
|
|
|
omar
Forum Guru
Arimaa player #2
Gender:
Posts: 1003
|
|
Re: Arimaa Challenge format for 2006
« Reply #11 on: Aug 17th, 2005, 3:29pm » |
Quote Modify
|
on Aug 17th, 2005, 11:17am, Fritzlein wrote:| I understand the resons for your decision Omar. I don't think the general public will perceive it in same way you do, but that's OK. |
|
Well actually it is very important for me that at least the active Arimaa community understands the reasons clearly and is supportive of it.
I really do regret having to make this adjustment to the challenge format after the initial challenge announcement. However I still firmly beleive that the change is still compatible with my original goal for the challenge. There is still a good chance that the challenge can be acheived through software advances before the hardware advances make it obsolete. Though the new format makes the challenge a little more difficult there will still be plenty of room between the best human players and perfect play for a program to demonstrate that it is better than any human player.
The bottom line is this. When I announced the challenge I forgot to consider two important factors.
1 Difficulty of getting the best human player to play a long match against the challenging program.
2 The "newness factor" of the game which favors computers because humans require a long time to master a game and the number of people playing the game will be small.
I realize now that the challenge I put out would have been fine for a well established game like Go with lots of history and players, but it is not suitable for a new game where there are no professional players and the player community is still germinating. I would hate to see the challenge lost due to one of these two factors. The new format tries to fix both of these oversights at once while still keeping the challenge acheivable in the way it should be acheived. Hopefully this format will work out and there won't be any need for more changes.
Quote:| This year you won't have to invite me to play: I'll be actively trying to qualify. I want to be part of the 9-0 demolition! |
|
Good luck. I will also be trying to make the team
|
|
IP Logged |
|
|
|
omar
Forum Guru
Arimaa player #2
Gender:
Posts: 1003
|
|
Re: Arimaa Challenge format for 2006
« Reply #12 on: Aug 17th, 2005, 4:10pm » |
Quote Modify
|
on Aug 15th, 2005, 6:43pm, 99of9 wrote:
Just for interest, was there any progress on this? |
|
With regard to soliciting sponsors to help increase the challenge prize. I do plan to still do that, but only after the format is definitely fixed.
One thing I was considering is allowing a sponsor to set their own time frame for the amount they are offering. For example a sponsor could say they are offering $5,000 until the year 2010. So the total challenge prize for the current year would then be $15,000.
|
|
IP Logged |
|
|
|
omar
Forum Guru
Arimaa player #2
Gender:
Posts: 1003
|
|
Re: Arimaa Challenge format for 2006
« Reply #13 on: Aug 17th, 2005, 5:07pm » |
Quote Modify
|
on Aug 15th, 2005, 10:38pm, Fritzlein wrote:
I do have a few minor suggestions, though. (Don't I always? ) I think that the effectiveness should not be calculated by 100*W/(W+L), as normal percentages are. A 50% score should be neutral, and anything below 50% should count against you. Therefore instead of percentage, we should use 100*(W-L)/(W+L). Right now my score of 3.25 could be beaten by someone who scores 7 of 15 against each bot. But having a losing record against the bots primarily shows that you are not fit to defend the challenge, even though you are practicing a lot.
|
|
Good suggestion. I've incorporated it.
|
|
IP Logged |
|
|
|
Fritzlein
Forum Guru
Arimaa player #706
Gender:
Posts: 5928
|
|
Re: Arimaa Challenge format for 2006
« Reply #14 on: Aug 17th, 2005, 10:05pm » |
Quote Modify
|
on Aug 17th, 2005, 3:29pm, omar wrote:
The bottom line is this. When I announced the challenge I forgot to consider two important factors.
1 Difficulty of getting the best human player to play a long match against the challenging program.
2 The "newness factor" of the game which favors computers because humans require a long time to master a game and the number of people playing the game will be small.
|
|
Those are very good points, and I can see why they motivate changing the challenge. Also I can see how they make Arimaa not comparable to Go, where there are full-time professional players and years of human understanding which computers are trying to catch up to.
Quote:
I would hate to see the challenge lost due to one of these two factors.
|
|
I quite understand that if a bot wins 5 games out of 9, or 2 mini-matches out of 3, that it doesn't mean Arimaa is easy for computers and can be solved for all practical purposes by brute force. The situation is not at all like chess, where a computer that surpasses humans is unlikely to later be overtaken. There probably aren't any startling things left to discover about chess that will allow creative thinkers to make a future surge relative to brute force thinkers.
It would indeed be a shame if computers won the Arimaa Challenge and everyone lost interest, but then later the humans discovered powerful new Arimaa strategies that again gave them the upper hand, and more permanently. It would also be a shame if some human Arimaa superstar emerged only after the challenge had already been lost, someone who then stayed ahead of computers his/her entire life. I see what you are trying to avoid, and I truly hope these scenarios never come about.
Unfortunately, I think that making the challenge more difficult to achieve will (A) not prevent the public from losing interest in Arimaa if a bot wins a simple majority in a match, and (B) send the signal that computers might be better than humans already without any AI breakthroughs. I understand the complexity of the situation, I don't think you will keep moving the goalposts to avoid giving the prize, and I hope the rest of the Arimaa community understands as well. Unfortunately I doubt that more than a few dozen people are interested in the reasons behind the headline. 
From a PR perspective, it might make more sense to strengthen the challenge in a way that won't make the headline. In fact, having a human team of three is great idea. It is not moving the goalposts as long as the victory condition remains a majority, and if anything the change emphasizes that humans in general, not just one human, are better than the bots. Furthermore, it should basically fix the problem of not having professional defenders of the challenge.
A more important way to keep the challenge strong is something you mentioned in another thread. You said:
on Aug 17th, 2005, 5:56pm, omar wrote:
Last year it was 20 rated games against at least 6 different opponents using the 2/2/100/10/8 time control. At least 6 of those games must be played against at least 3 different human opponents. Maybe I should just leave that part the same.
|
|
Certainly you shouldn't weaken this to only 2 human opponents and only 4 games as the page currently says. On the contrary, I think there is actually some small danger of humans losing a sweep to a bot that they've had little experience against, compared to almost no danger of losing 2 of 3 mini-matches to a bot they've been able to feel out more.
Just consider this past year. Do you remember the tension that Belbo might lose to the improved Bomb? We just didn't know at first, and indeed he struggled in the early games. But even by the end of the match Belbo had learned a lot, and won the last game without losing a piece or breaking a sweat. And then in the months since, we have discovered at least four ways to win mechanically, not to mention multiple critical strategic flaws in Bomb that won't be easy to patch even if the mechanical winning methods can be disrupted.
Bots do not, at the moment, hold up to human examination, which is a distinction of intelligence, which is precisely what the challenge is (or should be) getting at. The rules as you currently published them are more likely to result in loss of the challenge than it would be if you lowered the bar to 2 of 3 mini-matches, but stipulated that each defender be allowed two practice games against the bot's final version. You could even remove the human-play requirement during November and December and still have little to fear. I can't believe that, given the current state of affairs, we could lose after having those six practice games. On the contrary, the most likely way we could lose is by sheer surpise.
The "newness" factor was a great danger on the first challenge, but has become so less every year. The things we are now learning about the game are getting more subtle and harder to encode. It was much easier to deal with the elephant blockade than it will be to deal with the elephant-horse attack, for example. And that was last year's innovation! This year we have discovered how flooding one side (or both) can often be a good idea, but not always. It may be extremely difficult for a computer to judge when to attack all-out, as well as to know how to respond when a human does.
Quote:
I will also be trying to make the team
|
|
Then we'll have even less to worry about.
Anyway, I don't mean to be complaining, I'm just trying to give advice that I think will be good for Arimaa as a whole. Even when we disagree, your decisions always make sense to me, and they usually turn out to be better than what I would have done!
|
| « Last Edit: Aug 17th, 2005, 10:12pm by Fritzlein » |
IP Logged |
|
|
|
|
Home
Help
Search
Members
Login
Register
Reply
Notify of replies
Send Topic
Print