|
||||||||
|
Title: Arimaa Challenge format for 2006 Post by omar on Aug 15th, 2005, 4:37pm http://arimaa.com/arimaa/challenge/2006/ In another thread I had mentioned earlier this year that I would be changing the format for the Arimaa challenge match. See that thread for reasons: http://arimaa.com/arimaa/forum/cgi/YaBB.cgi?board=talk;action=display;num=1105700919 Another thing which I changed is the selection of the human players. I decided to look at the record of the players against the previous years bots to determine who the selected players will be. This makes the selection process less subjective. |
||||||||
|
Title: Re: Arimaa Challenge format for 2006 Post by 99of9 on Aug 15th, 2005, 6:43pm This post is about to get extremely harsh. I just want to say upfront that I admire the effort and generosity that you put into this game Omar, and I do not doubt your integrity, or motives for making this change. But I still want to say these things, because I think going ahead with this would be a blunder: I still think that it is a bad idea to change the challenge to make it much harder. Even if that was your original intent (although I for one did not know that it was until this year). Moving goalposts just don't instill confidence or trust that they will not move again every year. Your intent is so all-encompassing that it could be used to justify almost any increase in difficulty. Here is the intent you stated on the other thread, just so we don't have to refer back: Quote:
What if I were to propose a format where every human on the arimaa list played one game (with gold) against the bot, and if the bot lost a single game, it lost the challenge. I think that would be supremely unfair, but it could be justified as more likely to satisfy your intent. In fact even modern computer-chess programs would probably not win your proposed challenge format. To me, humans have clearly been surpassed in chess (not all areas of it, but as a whole), but they are still capable of winning one out of three 3-game matches. I thought we were claiming that arimaa was orders of magnitude more difficult for computers than chess? If that is really the case, then any defeat of humanity (in the old challenge format) would indicate that either we are wrong, or that AI has improved. If the challenge keeps getting harder, can we really say that AI is not making advances in leaps and bounds? No. One of the problems with getting AI accepted in the general community is that each time an intelligence problem is proposed, people say "no way a computer could do that - that requires true intelligence". Then some clever computer scientist achieves it, and the people say "that was just computation that you did, brute force, therefore that's not intelligent". So AI is forever on the horizon, and all the real advances are not credited as such, because the goals have moved. For what it's worth, I think some of the stuff that Jeff and Haizhi are doing is actually an AI advance, although admittedly not in as grand a way as you were initially anticipating. Quote:
I'd say that this modification is not purely a logistical change. Thus if you go ahead with this, the possibility of challenge-difficulty-altering changes should also be mentioned in the rules disclaimer. Quote:
Just for interest, was there any progress on this? :-) |
||||||||
|
Title: Re: Arimaa Challenge format for 2006 Post by jdb on Aug 15th, 2005, 7:49pm In my opinion, winning two out of three games against all three selected opponents seems fair enough to me. |
||||||||
|
Title: Re: Arimaa Challenge format for 2006 Post by Fritzlein on Aug 15th, 2005, 9:28pm I'm afraid I agree with 99of9 in thinking that the new challenge rules will be very detrimental. What catches the attention of the public is the basic idea, "Hey, there's a boardgame where humans can still beat the computers!" That's more important than the dollar amount of the prize, or the length of the match, or the structure of the match. The idea is that Arimaa is a game of real intelligence. That's what the Web site says, and that's the only reason most people pay any attention to Arimaa at all. Suppose that you offered a prize of one million dollars that no bot could beat the top human 10 games out of 10 in a match. That huge amount of money would be less impressive than the ten thousand dollar prize has been, because there is no emotional impact to saying "Computers are not totally infallible at Arimaa," or "Human intelligence is occasionally useful in Arimaa." Who cares? The gut-check question is whether man or machine is better. I can foresee the following disaster scenario with the new rules: Some year the bot will score 3-0, 1-2, and 3-0 against the human defenders respectively, for a total score which is a dominant 7-2 victory over the humans. Now, because of the way the match is structured, the prize money won't be awarded, but so what? Nobody will care. It will be all over in the minds of the public. There will be no headlines like "Humans regain dominance in Man vs. Machine" or "Humans can still outsmart computers". It's not interesting to anybody that "Humans are not totally humiliated at Arimaa yet." If that happens Arimaa will instantly lose all of its cachet anyway, so you might as well give the prize to the bot developer then. Admittedly, the old rules which pit the best computer against a single human didn't 100% precisely answer the question of whether Arimaa is a game of real intelligence, but that's what that match stood for to the masses. If you want to keep that flame of interest alive, I think the new structure has to give out the prize money as soon as computers are "better than humans". If there are three mini-matches against top humans, and if the computer wins two out of three of those mini-matches, then machine has already triumphed over man no matter what the rules say. Like 99of9, I must apologize for sounding harsh about your plans, Omar. Of course, since it is your game, your server, and your money, you can do whatever you like. You are legally and morally in position to have the Arimaa Challenge be what you want it to be. My point is only that if you change it the way you are proposing, it will take the air out of Arimaa's sails. It will detract from Arimaa's aura of being something special that humans are good at. |
||||||||
|
Title: Re: Arimaa Challenge format for 2006 Post by PMertens on Aug 15th, 2005, 9:48pm I absolutely agree with the former posts. In addition I would like to mention that there is absolutely no need to raise the bar for the bots. The current champion can not only be outplayed in one but in several different ways with all kinds of handicaps. Currently David would probably not even need to change anything to win again, but even if he does it is highly doubtful that we wont find an exploit before the challenge match. Depending on who will play this year you might even see a handicap game during the challenge. |
||||||||
|
Title: Re: Arimaa Challenge format for 2006 Post by Fritzlein on Aug 15th, 2005, 10:38pm on 08/15/05 at 16:37:28, omar wrote:
I like the new formula! This thing I like best about it is that we will now have a sort of bot-bashing contest between humans to see which of us will qualify to defend the honor of humanity. The people who win that contest will be people who really want to smash the championship bot, as opposed to people who have to be reluctantly pressed into service. I like that you key on the minimum number of wins across all four bots. It makes sense that you should be good against all bots to make the human team. I do have a few minor suggestions, though. (Don't I always? :-)) I think that the effectiveness should not be calculated by 100*W/(W+L), as normal percentages are. A 50% score should be neutral, and anything below 50% should count against you. Therefore instead of percentage, we should use 100*(W-L)/(W+L). Right now my score of 3.25 could be beaten by someone who scores 7 of 15 against each bot. But having a losing record against the bots primarily shows that you are not fit to defend the challenge, even though you are practicing a lot. My second suggestion is that for the four championship bots from last year, since they are being used as a qualifying benchmark, unrated mode should be suspended. Every game against those bots should be rated. In particular, it is possible to get a favorable initial piece setup in botbashing by repeatedly resigning games on the second move until the bot randomly places the pieces where you want them. Those early resignations, which currently abort the game with no record, should be forbidden (i.e. they should count as losses so no one will do it). The type of botbashing to qualify for the human team should be about consistent winning, not about the spectacular death-defying feat you can only get one time out of ten. And finally, I suggest that you wipe the slate clean and start everyone from scratch. I think that would be fairer to people who might want to qualify, but didn't know what the rules were going to be until now. Anyway, whatever the rules are for qualifying, this year I'm going to try hard to make the human team. I want to be part of the squad that kicks some bot behind. |
||||||||
|
Title: Re: Arimaa Challenge format for 2006 Post by 99of9 on Aug 16th, 2005, 6:54pm on 08/15/05 at 22:38:43, Fritzlein wrote:
I agree that aborted rated games should probably count (although it's probably safe to trust humans with that for the moment - I don't think the competition will be so intense that people resort to this kind of cheating. I'm not sure I agree with counting unrated games, because then no one will play handicap botbashes against those bots until a year after they are designed. |
||||||||
|
Title: Re: Arimaa Challenge format for 2006 Post by Fritzlein on Aug 16th, 2005, 8:44pm on 08/16/05 at 18:54:48, 99of9 wrote:
Oh, well, now that I think about it, allowing unrated games isn't so bad. If someone wants to practice a few times to improve technique before risking a rated game, why not? At least it would show that they were serious about the challenge. |
||||||||
|
Title: Re: Arimaa Challenge format for 2006 Post by PMertens on Aug 16th, 2005, 10:31pm on 08/15/05 at 22:38:43, Fritzlein wrote:
Well, you are currently leading the team ... and only 2 other persons qualify (but 2 more trying to get in ?) How about creating an official competition with a direct link from the main page ? |
||||||||
|
Title: Re: Arimaa Challenge format for 2006 Post by omar on Aug 17th, 2005, 2:56am Trying to find a human player for last years challenge match turned out to be more of a challenge than I had expected. I realize now that asking one person to play 8 intense games with the preasure of defending the challenge prize is a bit much. So the obvious solution was to have multiple players to distribute the games among. But once you have multiple humans player with each playing an equal number of games against the bot a new issue arises as to what the criteria should be for the bot to demonstrate that it is better than even the best human. The three obvious ones are: 1. The bot wins if it wins the majority of the games 2. The bot wins if it wins the individual series against the majority of the human players 3. The bot wins if it wins individual series against each of the human players The first two options do not change the difficulty of the challenge match, but the third option does. The more human opponents there are the more difficult it becomes. However, it is the only option that seems logical in demonstrating that the bot is clearly better than any and all humans. Karl pointed out in the other thread that if there were 4 human players then the chances of the bot winning would decrease to 6.25% if we use the third option. I mentioned in the thread that the Arimaa challenge as it currently stands is not as difficult as it should be and the new format helps to make it more difficult. I will restate the reasons again here in case anyone has not read that thread. There are several reasons why the current Arimaa challenge is so weak. First of all the most talented players in the world have probably not even discovered Arimaa yet. The population of Arimaa players is so low that the probability that the most talented player is actively playing Arimaa is less than one in a million. Secondly among the players that are actively playing Arimaa the probability that the best one will be selected to play in the challenge match is also fairly low. Finally it takes human many years to master complex strategy games. Even with hundreds of years of knowledge about the game and master level teachers available it takes even talented players many years of active playing and studying to reach a significantly high level of play. So even if the most talented players were actively playing Arimaa today, it would still take many years for them to achieve what would be considered a very high level of play in the game. Keep in mind that the current players are just playing for fun and not for a living. Thus even if Arimaa was a more difficult game for computers than Go, the task of developing a program to defeat the best Arimaa player is significantly easier than the task of developing a program to defeat the best Go player. Even with the new proposed format for the challenge match I believe it still remains easier. I also mentioned that I do need to be careful that the new format does not make the Arimaa challenge impossible to achieve within the scope of it's time frame. The hope is that if the Arimaa challenge is achieved, it will be done by significant advances made in software and not just due to the advances in hardware; thus the time limit of year 2020. When the challenge is made more difficult it means that the performance of the program must be significantly more than the performance of the human players to achieve it. Indirectly it raises the question of what is the level of perfect play and how close the humans will be to that level within the time frame of the challenge. Will there be enough room between the best human players and the limit of perfect play for a program to demonstrate that it is clearly better? I think that within the time frame of the challenge, Arimaa will still be such a new and unexplored game that there will be sufficient room for a program to demonstrate that it has achieved a significantly higher level of play than the humans. There might be concern that the challenge rules may be changed again in the future to make the challenge more difficult. Although I was careful to leave that option open in the early years by adding the disclaimer, I want to remove that option completely and the challenge rules should be practically etched in stone. I would like to remove that option next year if things go smoothly this year and there is no problem in finding players. |
||||||||
|
Title: Re: Arimaa Challenge format for 2006 Post by Fritzlein on Aug 17th, 2005, 11:17am I understand the resons for your decision Omar. I don't think the general public will perceive it in same way you do, but that's OK. With luck, the humans will score a 9-0 shutout victory this year to keep alive among the masses the idea that Arimaa is hard for computers to play well. Furthermore, if the gap between humans and computers continues to widen (as it seems to be doing at the present) there will be more shutouts in the years to come. If that happens, the exact criteria for winning the challenge become less important. on 08/17/05 at 02:56:07, omar wrote:
I don't know how other people feel, but the new rules make a huge difference in my eagerness to play. It is hard to play four games per week for two weeks in a row if one has a full-time job. But now, if the challenge is spread across three weeks, each human only has to play one game per week. (The bot presumably doesn't get tired, so it can play three times per week.) Playing one official game per week should be no problem at all, even if work is busy. This year you won't have to invite me to play: I'll be actively trying to qualify. I want to be part of the 9-0 demolition! :-) |
||||||||
|
Title: Re: Arimaa Challenge format for 2006 Post by omar on Aug 17th, 2005, 3:29pm on 08/17/05 at 11:17:47, Fritzlein wrote:
Well actually it is very important for me that at least the active Arimaa community understands the reasons clearly and is supportive of it. I really do regret having to make this adjustment to the challenge format after the initial challenge announcement. However I still firmly beleive that the change is still compatible with my original goal for the challenge. There is still a good chance that the challenge can be acheived through software advances before the hardware advances make it obsolete. Though the new format makes the challenge a little more difficult there will still be plenty of room between the best human players and perfect play for a program to demonstrate that it is better than any human player. The bottom line is this. When I announced the challenge I forgot to consider two important factors. 1 Difficulty of getting the best human player to play a long match against the challenging program. 2 The "newness factor" of the game which favors computers because humans require a long time to master a game and the number of people playing the game will be small. I realize now that the challenge I put out would have been fine for a well established game like Go with lots of history and players, but it is not suitable for a new game where there are no professional players and the player community is still germinating. I would hate to see the challenge lost due to one of these two factors. The new format tries to fix both of these oversights at once while still keeping the challenge acheivable in the way it should be acheived. Hopefully this format will work out and there won't be any need for more changes. Quote:
Good luck. I will also be trying to make the team :-) |
||||||||
|
Title: Re: Arimaa Challenge format for 2006 Post by omar on Aug 17th, 2005, 4:10pm on 08/15/05 at 18:43:58, 99of9 wrote:
With regard to soliciting sponsors to help increase the challenge prize. I do plan to still do that, but only after the format is definitely fixed. One thing I was considering is allowing a sponsor to set their own time frame for the amount they are offering. For example a sponsor could say they are offering $5,000 until the year 2010. So the total challenge prize for the current year would then be $15,000. |
||||||||
|
Title: Re: Arimaa Challenge format for 2006 Post by omar on Aug 17th, 2005, 5:07pm on 08/15/05 at 22:38:43, Fritzlein wrote:
Good suggestion. I've incorporated it. |
||||||||
|
Title: Re: Arimaa Challenge format for 2006 Post by Fritzlein on Aug 17th, 2005, 10:05pm on 08/17/05 at 15:29:42, omar wrote:
Those are very good points, and I can see why they motivate changing the challenge. Also I can see how they make Arimaa not comparable to Go, where there are full-time professional players and years of human understanding which computers are trying to catch up to. Quote:
I quite understand that if a bot wins 5 games out of 9, or 2 mini-matches out of 3, that it doesn't mean Arimaa is easy for computers and can be solved for all practical purposes by brute force. The situation is not at all like chess, where a computer that surpasses humans is unlikely to later be overtaken. There probably aren't any startling things left to discover about chess that will allow creative thinkers to make a future surge relative to brute force thinkers. It would indeed be a shame if computers won the Arimaa Challenge and everyone lost interest, but then later the humans discovered powerful new Arimaa strategies that again gave them the upper hand, and more permanently. It would also be a shame if some human Arimaa superstar emerged only after the challenge had already been lost, someone who then stayed ahead of computers his/her entire life. I see what you are trying to avoid, and I truly hope these scenarios never come about. Unfortunately, I think that making the challenge more difficult to achieve will (A) not prevent the public from losing interest in Arimaa if a bot wins a simple majority in a match, and (B) send the signal that computers might be better than humans already without any AI breakthroughs. I understand the complexity of the situation, I don't think you will keep moving the goalposts to avoid giving the prize, and I hope the rest of the Arimaa community understands as well. Unfortunately I doubt that more than a few dozen people are interested in the reasons behind the headline. :-( From a PR perspective, it might make more sense to strengthen the challenge in a way that won't make the headline. In fact, having a human team of three is great idea. It is not moving the goalposts as long as the victory condition remains a majority, and if anything the change emphasizes that humans in general, not just one human, are better than the bots. Furthermore, it should basically fix the problem of not having professional defenders of the challenge. A more important way to keep the challenge strong is something you mentioned in another thread. You said: on 08/17/05 at 17:56:07, omar wrote:
Certainly you shouldn't weaken this to only 2 human opponents and only 4 games as the page currently says. On the contrary, I think there is actually some small danger of humans losing a sweep to a bot that they've had little experience against, compared to almost no danger of losing 2 of 3 mini-matches to a bot they've been able to feel out more. Just consider this past year. Do you remember the tension that Belbo might lose to the improved Bomb? We just didn't know at first, and indeed he struggled in the early games. But even by the end of the match Belbo had learned a lot, and won the last game without losing a piece or breaking a sweat. And then in the months since, we have discovered at least four ways to win mechanically, not to mention multiple critical strategic flaws in Bomb that won't be easy to patch even if the mechanical winning methods can be disrupted. Bots do not, at the moment, hold up to human examination, which is a distinction of intelligence, which is precisely what the challenge is (or should be) getting at. The rules as you currently published them are more likely to result in loss of the challenge than it would be if you lowered the bar to 2 of 3 mini-matches, but stipulated that each defender be allowed two practice games against the bot's final version. You could even remove the human-play requirement during November and December and still have little to fear. I can't believe that, given the current state of affairs, we could lose after having those six practice games. On the contrary, the most likely way we could lose is by sheer surpise. The "newness" factor was a great danger on the first challenge, but has become so less every year. The things we are now learning about the game are getting more subtle and harder to encode. It was much easier to deal with the elephant blockade than it will be to deal with the elephant-horse attack, for example. And that was last year's innovation! This year we have discovered how flooding one side (or both) can often be a good idea, but not always. It may be extremely difficult for a computer to judge when to attack all-out, as well as to know how to respond when a human does. Quote:
Then we'll have even less to worry about. :-) Anyway, I don't mean to be complaining, I'm just trying to give advice that I think will be good for Arimaa as a whole. Even when we disagree, your decisions always make sense to me, and they usually turn out to be better than what I would have done! |
||||||||
|
Title: Re: Arimaa Challenge format for 2006 Post by clauchau on Aug 18th, 2005, 9:48am I like the change Omar is making. I see flaws in any format, but I like the new flaws slightly better :) Well, I have subjective reasons. When I think how inconsistent and so totally subject to pressure I can be, while we know bots still are stronger on that regard, I feel released for the human players. I'm not satisfied with the match between Deep Blue and Kasparov. Chess AI in general almost fulfilled its dreams and mine, at least statically. But what with Kasparov big act, his high level as a strategist in real life, the million he got, the chess experts unsettled? I know it's bad to question a past event and Kasparov, whom I am fond of. But about that event, I can't help. Asking for a panel of players instead of one single judge, although less of a drama, gets us closer to collective and future acceptance of the result. The Loebner Prize awarding the first AI passing the Turing Test has a panel of judges too. on 08/17/05 at 02:56:07, omar wrote:
Well, human aren't that brilliant then and Arimaa isn't that interesting a challenge for AI? ::) |
||||||||
|
Title: Re: Arimaa Challenge format for 2006 Post by clauchau on Aug 18th, 2005, 11:35am on 08/15/05 at 18:43:58, 99of9 wrote:
By the way Arimaa seems to me now easier a game than I thought in the beginning. It is complex, harder than Chess, but BotBomb is not that lost at the elementary level I thought early bots would. In any case I still think Arimaa is challenging to AI, but maybe only of "one" order of magnitude. |
||||||||
|
Title: Re: Arimaa Challenge format for 2006 Post by clauchau on Aug 18th, 2005, 12:33pm on 08/17/05 at 11:17:47, Fritzlein wrote:
Indeed that's not fair in case the bot needs as much time as humans to think over the past games and evolve before the next game. I hope AI will get to that point soon. In that case the bot shall be allowed as many running clones as there are human opponents. Hey, about being fair, the clones shall be kept informed of every recent Arimaa games. And how about letting the bot know its opponents' identities on each game? Humans know whom they are playing against. |
||||||||
|
Title: Re: Arimaa Challenge format for 2006 Post by omar on Aug 18th, 2005, 6:15pm on 08/18/05 at 12:33:31, clauchau wrote:
The bots are allowed to run between games to analyze the game. Also the Arimaa games archive is available for download so bots can make use of the recent games. The name of the opponent is available to the bot as part of the game state. |
||||||||
|
Title: Re: Arimaa Challenge format for 2006 Post by omar on Aug 18th, 2005, 6:48pm on 08/17/05 at 22:05:18, Fritzlein wrote:
I've always thought that as long as there was even one human player who could defeat the computer, then humanity wins. I think that the general public looks at it that way too. Suppose Kasparov, Anand and Kramnik played Deep Blue (or some strong chess program) and the result was: Anand: 0 0 0 Kramnik: 0 0 0 Kasparov: 1 1 1 The headlines would go crazy about how Kasparov defended humanity. No one would care that Anand and Kramnik got demolished. I think the public looks at each players record independent of others. They would think that Anand and Kramnik aren't as good, but Kasparov is certianly better. The public doesn't care about the average performance of the humans; they just want to know if there is at least one that is better. Suppose the result turned out as: Anand: 0 0 0 Kramnik: 1 1 0 Kasparov: 1 1 0 And the computer is declared the winner since it won more total games. Now have fun trying to convince the average person that humans aren't better than the computer :-) |
||||||||
|
Title: Re: Arimaa Challenge format for 2006 Post by Janzert on Aug 18th, 2005, 7:45pm If I could interject a few words as an interested spectator. Of course having no other qualifications, feel free to ignore it. Personally what I would find most interesting is simply entering computers into the world championship along side, and under the same rules as, humans. Especially once, as I imagine will happen within the next few years, a minimum rating is required to enter. In this regard I was happy to see bomb included in the recent postal tournament. But in regards to the challenge I can understand the practicality of a stand alone event in terms of excitement, PR, etcetera. One of the things that has always bothered me about human vs. machine chess events is the frequent imbalance of game history available to the two sides. Almost always the computer team will have thousands of past games to study from the human opponent. Whereas usually the human has very few, often less than a hundred from the current version or even total. I think this may be even more detrimental in Arimaa where exploration is still continueing in basic strategies. So both lowering the prerequisite number of games and having 3 game matches (even though there are 3 of them) makes me very nervous. I would much rather see an expansion in the prerequisite games rather than any reduction. I can understand time requirements not allowing longer matches, in return it would be nice to have some optional practice games to help make up for this though. Maybe ideally something like the two weeks before the challenge the bot is available for reservation by the challengers to play games against. These changes would make it harder for the computer to win. But I'm afraid a not unlikely scenario with the current rules is for the humans to lose the challenge when presented with a novel strategy, resulting in a loss of interest. But then the same bot to be regularly and easily beaten shortly afterwards when it's flaws are discovered. Janzert |
||||||||
|
Title: Re: Arimaa Challenge format for 2006 Post by Fritzlein on Aug 18th, 2005, 8:52pm on 08/18/05 at 18:48:19, omar wrote:
You may be right about public perception. I guess we won't know until word about the new challenge format gets around (probably not before the challenge is actually played again) and the reactions start to trickle in. |
||||||||
|
Title: Re: Arimaa Challenge format for 2006 Post by Fritzlein on Aug 20th, 2005, 10:08pm on 08/17/05 at 16:10:10, omar wrote:
If I were a sponsor I would want the rules to fixed in stone before putting up any money, so this seems like a good idea. Quote:
What are you willing to give in exchange for sponsorship? I'll bet you could get $100,000 if you were willing to go all the way with the sponsor on a 15-year contract. Intel, for example, is a likely candidate to put up the money, if you would do things such as 1) Rename it the "Intel Arimaa Challenge" and agree to refer to it that way on all your Web pages and press releases 2) Agree to have the challenge bot play on an Intel chip every year, so that if the challenge is awarded, Intel also wins (i.e. it would a bot running on Intel which beat down the humans) 3) Have a banner advertisement to Intel from the arimaa.com homepage, the gameroom, and all the other pages in arimaa.com. Perhaps some corporations want to be associated with something just because it is cool, but probably to get any serious interest you would have to splash their name everywhere on any property you control. I personally wouldn't mind if Arimaa were "sold out" in this way, because the increase in prize money and the backing of a major corporation would bring a bunch of new attention to the game. I could handle looking at a few advertisements if it helps the game take off even more than it already has for such a young game. Whatever it takes to spread the word, I'm in favor. (By the way, Intel briefly was a big sponsor of chess until Kasparov bucked them to play Deep Blue. Obviously Intel was not pleased that IBM got all the good press, and Intel quitely totally withdrew their sponsorship. Despite being burned in that way, Intel might still be interested in computer gaming.) Another thing to propose to make it seem less risky to the sponsor is to ask them to chip in $10,000 per year for each year that the challenge isn't won. Then they don't have to worry about putting up a huge amount of money for only one or two years of exposure. |
||||||||
|
Title: Re: Arimaa Challenge format for 2006 Post by 99of9 on Aug 21st, 2005, 4:19am on 08/21/05 at 02:43:50, Arimanator wrote:
Of course you are right that we should be careful about what we say about potential sponsors, but if it helps, I don't think Fritz's comments are mocking at all. |
||||||||
|
Title: Re: Arimaa Challenge format for 2006 Post by Fritzlein on Aug 21st, 2005, 8:56am on 08/21/05 at 02:43:50, Arimanator wrote:
I'm not sure what part of my post seems to be mocking Intel; perhaps they would freely admit that their motivation for sponsoring Arimaa would be advertising exposure more than just backing something cool. And I wasn't trying to laugh at them for being burned by Kasparov. On the contrary my sympathies are rather with Intel than the Professional Chess Association, and it was in my mind when I included that post-script that it is simply good for a business relationship to respect what your sponsor wants, as Kasparov failed to do. On the other hand, I admit that I was writing for a local audience and didn't consider how it might read to a potential sponsor. It's good to remember that the forum is public, and the folks at Intel (or anywhere) can read it too. Thanks for that reminder, and I'll try to imagine someone from Intel reading each of my posts before I submit them in the future. Quote:
Yes, it is precisely because advertising is so expensive that I think the money is out there for Arimaa. While I'm rambling on the subject, I should mention that if Intel were the sponsor (or Dell or Hewlett Packard or anyone who would want the challenge bot running on their brand of hardware) they might actually want the challenge money to be awarded, because it would mean that their hardware would win. So among the things we could give to Intel would be 4) Allow the bots to run on as fast a machine as Intel will provide us for free, and mention everywhere what an awesomely powerful computer that is (e.g. the bot is considering more nodes per second on an Intel system than Deep Blue was against Kasparov!). Imagine how good that would make a hardware sponsor feel. :-) I know that this runs contrary to Omar's intentions that the challenge be won by better software rather than faster hardware, but I'm just trying to look at it from the perspective of the sponsor. A computer hardware sponsor would want the hardware to play as large a role as possible, and they might be hoping for humanity to lose even if that meant forking over $100,000 they could otherwise keep. So it gets back to the question of how much Omar is willing to cooperate with the sponsor in exchange for money and other support. To drum up Intel's enthusiasm, we'd probably have to make it Intel's game. But if we had their enthusiasm, they might do more than just sponsor the challenge, they might also give free hardware for the Arimaa server and as prizes for the Computer Championship matches, etc. It's a tradeoff that could have large benefits for the Arimaa community, although it would require some faith from Omar that a 100-fold hardware speedup wouldn't be enough to put the bots ahead of the humans. :-) |
||||||||
|
Title: Re: Arimaa Challenge format for 2006 Post by Arimanator on Aug 21st, 2005, 9:54am on 08/21/05 at 08:56:46, Fritzlein wrote:
To be perfectly honest I wouldn't stake my life on it ;D, especially if the bots are ones that never played humans in the open before and harbor some hidden new improvements. |
||||||||
|
Title: Re: Arimaa Challenge format for 2006 Post by 99of9 on Aug 21st, 2005, 9:59pm on 08/21/05 at 08:56:46, Fritzlein wrote:
Now imagine how good it would make a hardware sponsor feel if their off-the-shelf hardware were able to consider more nodes per second than Deep Blue! Even that should be possible by 2020, shouldn't it? |
||||||||
|
Title: Re: Arimaa Challenge format for 2006 Post by 99of9 on Aug 21st, 2005, 10:03pm on 08/21/05 at 08:56:46, Fritzlein wrote:
Supercomputers nowadays have many more than 100-fold better performance than a PC... |
||||||||
|
Title: Re: Arimaa Challenge format for 2006 Post by omar on Aug 22nd, 2005, 10:05pm on 08/20/05 at 22:08:23, Fritzlein wrote:
I actually did call Intel and ADM and talked to the heads of their marketing departments shortly after releasing Arimaa to see if they might be interested in sponoring the challenge. I found out that big companies don't like to sponsor things that aren't already well known and established. It's the chicken or the egg kind of situation. |
||||||||
|
Title: Re: Arimaa Challenge format for 2006 Post by omar on Aug 22nd, 2005, 10:45pm on 08/21/05 at 22:03:24, 99of9 wrote:
I would say the top supercomputer is maybe even 10,000 fold better performance than a PC. There is already a difference of 100 fold between the high end supercomputers and the low end supercomputers and the low end supercomputers have 100's of processors that used in todays PCs. See: http://www.top500.org/lists/plists.php?Y=2005&M=06 Im pretty sure that a supercomputer running an Arimaa program could definitely beat todays players. I would not have been so sure about it when I was first releasing Arimaa, since at the time I was not considering the "newness" factor. But the point of the Arimaa challenge is not to encourage building something that can only be accquired by national labortories and big companies or which can easily disappears after the demonstration. Rather I would like to use the challenge to encourage work on AI software. I am a strong beleiver that even the ordinary PC's of today already have enough power to beat even the best human players of 2020 if given the right software. Just that it might take a lot of computing power to produce that software :-) |
||||||||
|
Title: Re: Arimaa Challenge format for 2006 Post by omar on Aug 23rd, 2005, 1:03am I was thinking again about just how much harder the new format is for the computers compared to the old format. Karl had pointed out in another thread that if a bot had a 50% chance of winning with the old format then it would only have a 6.25% chance of winning with the new format if there were 4 human players. Likewise the chance of winning would be 12.5% if there were 3 human players. I started wondering how much higher a bots rating would have to be above the humans rating to bring the bots chance of winning back up to 50%. If this number turned out to be like a 1000 or 2000 points then I might have made the challenge impossible to acheive and should reconsider the format. But my calculations are saying that a bot needs to only be about 155 rating points above the human to bring it's chance of winning back up to 50%. If that's the case then the new format is very reasonable. There is pleanty of room between the best player and perfect play for a bot to demonstrate that it is better than the best player. Can someone else try to see what number they come up with so we can verify this. Thanks. |
||||||||
|
Title: Re: Arimaa Challenge format for 2006 Post by Fritzlein on Aug 23rd, 2005, 6:06am If we assume that all 3 players are rated X, then by my calculation, the computer needs a rating of about X+153.7 to have 50% chance of sweeping the humans as three best-of-three matches. I can see why you say that 154 rating points is not very much, but the difference seems pretty significant to me. For chess we had approximate parity in in 1999 and now six years later computers might be getting to 154 points better with Hydra. For a challenge with only fifteen years left, six years is a fairly hefty chunk of time. :-) I understand, though, that such a gap in strength might mean more in chess where draws are more frequent. |
||||||||
|
Title: Re: Arimaa Challenge format for 2006 Post by Fritzlein on Aug 23rd, 2005, 6:22am on 08/22/05 at 22:45:47, omar wrote:
Are you talking about today's software as well as today's human players? I'm not as confident about that increase in play strength from faster hardware as you are. Remember how easy it was to beat Bomb in the postal giving it maybe a 20 to 1 time advantage? The weakness of strategic judgement in Bomb may cripple it even at great search depth. However, I concede a difference between a factor of 20 and a factor of 10,000. :-) I read that for chess there seems to be a rough correspondence between a doubling of think time and gaining 100 rating points. Now if Arimaa is as computer-resistant as it was designed to be, for Arimaa a doubling of hardware speed might result in a gain of only 20 rating points. Consider the difference in rating between Arimaazilla and Arimaazon, where the latter has 3 or 4 doublings of think time. Now 10,000 is about 13 doublings, so maybe 260 rating points? I still have more than that large a cushion above the current Bomb, and thus might withstand a 10,000-fold speed increase. I guess, given that we don't have a supercomputer, it is idle speculation, but interesting nonetheless. :-) |
||||||||
|
Title: Re: Arimaa Challenge format for 2006 Post by Fritzlein on Aug 23rd, 2005, 7:33am on 08/22/05 at 22:05:24, omar wrote:
Oh, that's too bad. I shouldn't be surprised, but I admit I am disappointed. Omar, I apologize if my previous posts implied that you weren't thinking of the obvious sponsors and the things you could offer them. I don't know much about marketing, and I was just speaking off the top of my head. You've done a pretty darn good job of marketing Arimaa all by yourself, without the help of corporate sponsorship. I don't think you have to do anything different for Arimaa to grow apace. The friendly server and terrific game you have created are already enough of a draw to keep the community growing steadily. |
||||||||
|
Title: Re: Arimaa Challenge format for 2006 Post by omar on Aug 23rd, 2005, 9:54am on 08/23/05 at 06:06:28, Fritzlein wrote:
Thanks for double checking that Karl. Also with Chess being such a well studied game with so many good player, every rating point in the space above the best human players is very hard to gain. So 154 points above the the best players in Chess is very significant compared to acheiving the same in Arimaa where the knowledge of the game is relatively unexplored and the level of play is probably not even at a Masters level yet. |
||||||||
|
Title: Re: Arimaa Challenge format for 2006 Post by omar on Aug 23rd, 2005, 10:30am on 08/23/05 at 06:22:24, Fritzlein wrote:
Probably even with today's software. Although the recent advances made in bot bashing within the last week or so by Naveed and Pat seem to be turning everything upside down. It's going to be interesting to see if the bots will be able to fix those holes. |
||||||||
|
Title: Re: Arimaa Challenge format for 2006 Post by omar on Aug 23rd, 2005, 11:08am on 08/23/05 at 07:33:03, Fritzlein wrote:
Oh don't worry about it; the thought never even crossed my mind. I would much rather have Arimaa be a game that grows slowly and steadily than have it suddenly explode into popularity. Remember it took the Internet almost 20 years before it reached a thousand nodes. But that slow early growth allowed it to build a solid infrastructure for the future. http://www.zakon.org/robert/internet/timeline/ Scroll down to the bottom to see the growth charts. Of course Arimaa will never see that kind of growth, but what I mean to say here is that I think its better to focus on building a solid infrastructure and not worry about growth so much. |
||||||||
|
Title: Re: Arimaa Challenge format for 2006 Post by omar on Aug 24th, 2005, 5:36am The new agressive techniques for winning against the bots are working great. But a side effect of that is that the selection process I was planning to use becomes sort of useless because a person who learns those techniques can pretty much go on winning as much as they want against the *2005CC bots. So I guess I might have to go back to just selecting the players directly. I would have liked the selection process to be directly related to how a player performs against the bots. Anyone have any suggestions for this situation. |
||||||||
|
Title: Re: Arimaa Challenge format for 2006 Post by Arimanator on Aug 24th, 2005, 6:18am on 08/24/05 at 05:36:20, omar wrote:
One solution to avoid people rehearsing a recipe to beat one particular bot would be to impose an (unusual) piece setting for the human player, one different for each player (to avoid copy cats ;) ) , with no time to prepare and see how each of us manages with it. It would have the advantage of selecting a person on his/er ability to beat bots more than humans and as I said one can't resort to recipes, at least not easily, when one doesn't know where the main pieces will be at the start of the game, let alone how the bot will place its pieces in response to that. |
||||||||
|
Title: Re: Arimaa Challenge format for 2006 Post by Fritzlein on Aug 24th, 2005, 7:08am on 08/24/05 at 06:18:22, Arimanator wrote:
That's an interesting idea. To make it simple one could impose the condition that the setup of the pieces for the human player is completely scrambled by the server before each game. Then every player would have to fight through terrible initial setups as well as the reasonable ones. Omar, I never had any objection to your original method of choosing the human defenders, i.e. arbitrarily choosing whomever you want. I still think that may be the best plan. I don't see why there has to be any more structure than that. But if you do want some objective criteria that have something to do with how well a human is likely to perform against an unknown bot next February, then it seems relevant to be able to beat all of last year's bots from a randomized starting position. There may still be methods that reliably beat each bot, but at least they can't possibly be move-for-move copies: the winning methods will then indisputably deserve to be called strategies. Perhaps, though, this criterion will still be too easy. Whatever criteria you settle on, if it involves beating bots repeatedly, might be something that too many humans can accomplish. That, in turn, might indicate that many folks are capable of defending the challenge at present, and humanity is not in any danger from the bots for the foreseeable future, so it really doesn't matter who defends the challenge. :-) |
||||||||
|
Title: Re: Arimaa Challenge format for 2006 Post by jdb on Aug 24th, 2005, 8:11am Quote:
Maybe something that involves the bots that attempt to qualify for the 2006 Computer Championship? |
||||||||
|
Title: Re: Arimaa Challenge format for 2006 Post by Arimanator on Aug 24th, 2005, 8:21am on 08/24/05 at 08:11:11, jdb wrote:
I am afraid that that would let "the cat out of the bag" prematurely and some might say that the world official contestant had got some tips out of these games and that the deck was stacked in favor of the human player. |
||||||||
|
Title: Re: Arimaa Challenge format for 2006 Post by Fritzlein on Aug 24th, 2005, 6:21pm Just to see what it would be like, I tried playing Bomb2005CC with my own pieces randomized. You can see the result in game 18622. Based on this one experience, I would say that randomizing the opening position definitely gets around the problem of formulaic wins, and requires one to show real strategic superiority over the bot to win. Nothing went as smoothly as planned for me. I didn't have time to launch the E+H attack before Bomb had attacked one of my home traps with E+D. Such an attack from Bomb was feasible because my trap had only dogs and rabbits next to it. Then I wanted to take the attacking dog hostage, but since I had rabbits on the second row, Bomb delayed the hostage-taking by pulling a rabbit into the trap. In order to finally get the hostage, I had to use my camel instead of a horse, because my random setup had put both horses on the other side, but this exposed my camel more than I would have liked. Even once I had gotten the upper hand (HDC for M), my further progress was hindered by the lingering effects of the original setup. My second-rank rabbits on the east flank made it much more difficult for me to get Bomb's camel hostage into position, and by the time the rabbits had cleared out, my two stupid corner cats (left there from the start) weren't enough to hold up the camel on h6. Eventually, after capturing the hostage camel for HR, my mass of central rabbits was a liability as much as an asset. And finally, in the endgame it hurt my goal defense and made things scarier to have my cats on the same side instead of split. On the other hand, Bomb still over-valued the camel and the camel hostage, still made reversible moves, still played too passively, and still showed a poor understanding of trap control at times, e.g. around f3 late. These are strategic weaknesses that undermine Bomb's play, and probably will continue to do so next year, since they are not easy to fix. It is my guess that anyone who can consistently beat all the 2005CC bots with their own pieces randomized at the start will be qualified to defend the next challenge. |
||||||||
|
Title: Re: Arimaa Challenge format for 2006 Post by 99of9 on Aug 24th, 2005, 10:59pm Interesting experiment - it definitely sounds harder. I'm a bit worried because in some ways this is a different game (like Fisher random vs chess). It might favour people who are not very good at traditional arimaa opening theory, which might mess them up when they get into the challenge against a good bot. But anyway, I think this solution is better than the original metric that Omar proposed. Of course I think it is also fine if Omar chooses participants directly. |
||||||||
|
Title: Re: Arimaa Challenge format for 2006 Post by Adanac on Aug 25th, 2005, 12:46pm on 08/24/05 at 22:59:33, 99of9 wrote:
One of the things that attracted me to Arimaa and away from chess is the idea that every opening is unique and that there's room for original thought from the very beginning of the game. Now that most games more or less have the same opening setup and similar opening tactics, and bots can be defeated repeatedly using the same techniques every times, I think it would be great to mix things up a bit. I'm speculating here, but perhaps these random opening positions would closer to the spirit that Omar had in mind when he created a game with a variable opening setup ?? By the way, I'm also I fan of Fischer Random Chess so I am a bit biased :) |
||||||||
|
Title: Re: Arimaa Challenge format for 2006 Post by Arimanator on Aug 25th, 2005, 1:01pm on 08/25/05 at 12:46:05, Adanac wrote:
Not to mention that we may discover things that our sheeplike attitude of immitating each other's openings has kept away from us. |
||||||||
|
Title: Re: Arimaa Challenge format for 2006 Post by omar on Aug 25th, 2005, 7:20pm Yeah, I never thought that we would gravitate towards a few opening setups. Maybe we aren't experimenting enough or maybe it's bound to happen regardless. I think it will take some time to get a better idea of where things will go. If we find that just a few setups are still dominating in a few years then I will definitely consider randomizing the opening setup. I think I've also mentioned before that if we find that gold has a noticable advantage due to first move, I will consider limiting golds first move to two steps. Making decisions on such adjustments to the rules just isn't possible until you have a massive collection of real games to look at. For the selection of the human players, the idea of requiring random setup is intersting, but what if one player happens to get really bad setups and another gets good ones. It does add a little randomness to the selection process. Also Im begining to think that relying only on a players performance against the bots might not be a good thing. How they perform against humans is also important incase next years bots are very different than the previous years bots. So for now I'll keep the selection process for the human players the same as before. But I will be taking the performance against last years bots into consideration. |
||||||||
|
Title: Re: Arimaa Challenge format for 2006 Post by Fritzlein on Sep 2nd, 2005, 10:23pm on 08/25/05 at 13:01:22, Arimanator wrote:
Fortunately, there are rewards for breaking the mold, even in competitive play. For example, in the first game of the final of the 2005 World Championship, I set up with Ra7, Ra8, Rc7, Rc8, Rf7, Rf8, Rh7, Rh8, a structure which is probably unsound and seldom played today, but which I believe had a psychological impact on Belbo. Nowadays I am probably one of the least experimental players, but fortunately there are still original thinkers out there who refuse to conform. JDB often starts with an elephant on a wing, and blue22 puts the elephant behind a trap. Naveed puts rabbits behind his traps instead of on the wings. And variations in the opening position are just the beginning of questioning accepted opening theory: we're also seeing lots of "unsound" swarming attacks. If super-aggressive play turns out to be effective, it will forcibly re-write the way we do our opening setups. In short, although I'm mostly too chicken to play originally myself, I'm happy to note that many other people are not, and glad that I might be wrong about everything. :-) |
||||||||
|
Arimaa Forum » Powered by YaBB 1 Gold - SP 1.3.1! YaBB © 2000-2003. All Rights Reserved. |