Author |
Topic: 2007 qualifying for computer championship (Read 4101 times) |
|
Fritzlein
Forum Guru
Arimaa player #706
Gender: 
Posts: 5928
|
 |
2007 qualifying for computer championship
« on: Nov 11th, 2005, 5:44pm » |
 Quote  Modify
|
Omar,
It's probably too late to change the rules for how bots qualify for the 2006 championships, but for next year I think a different qualifying process would be in order. Under the current rules bots have to expose themselves online for 40 days. Not only is that a long time for the developer to dedicate a special-purpose machine, it also is a long time for opponents to figure out exactly how to exploit any weaknesses.
Now I'm all in favor of humans having a fair shot at learning something about a bot if that bot is going to be playing for the Arimaa Challenge. But that's different from letting the all the other bot developers know everything about your bot. If I were an unscrupulous bot programmer, I would test out winning lines against specific bots that aren't random enough. I could have my bot think on its own if it got to a unique position, but if it were in a known position my bot would play a known winning line I tested out during those 40 days.
This vulnerability means that bot developers have every incentive to make changes at the last minute, so their bot doesn't play the same way in the championship as it did while on-line. You don't forbid developers from changing the bot during the 40 days (on the contrary, I expect you would welcome continuing development) so it isn't against the rules to make last-minute changes. And the incentive is all towards making any adjustments as late as possible.
The consequence of rules that encourage last-minute changes is to undermine the original point of the 40 days on-line. Humans could get familiar with a bot, and then be shocked to discover the same bot playing differently in the challenge match. I think the qualifying rules should totally change, so that there is no incentive for a bot to play weakly or differently in the run-up to the computer championship.
My idea is this: Let anyone who wants to participate and who has a working bot submit the code by December 31, 2006. Make those bots available for on-line play for two weeks before the tournament, but after the code is finalized. So for the first two weeks of 2007, humans would have a shot at Bomb2007CC, Clueless2007CC, etc. That would give the bots some ratings to seed them into the computer championship, which would then start on January 15. Also, it would preserve fairness between the bot developers, because no developer would have to put up their bot for the others to scope out ahead of time. Finally, it would make the challenge defense fair, because the humans would have a bit of practice against the bot they will actually be playing in the Challenge, as opposed to a bot that might be modified later.
Actually, in my opinion, two weeks of practice against all the bots is way more than necessary. The humans couldn't be taken by surprise even if only the three humans defending the challenge match had the chance to practice, and even if they only got three practice games each. But I'm erring on the side of caution here: maybe in some future year that won't seem like enough time.
I want to stress that I don't think any current developers are intentionally putting up brain-dead versions of their bots. I'm sure Bomb's present weakness has to do with a registration code or some such, and I'm sure Clueless' present weakness has to do with new features that aren't properly tuned yet. I'm just saying that we should totally remove the current incentive for developers to make their bots play weakly on purpose.
|
| « Last Edit: Nov 11th, 2005, 5:48pm by Fritzlein » |
 IP Logged |
|
|
|
99of9
Forum Guru
Gnobby's creator (player #314)
 
Gender:
Posts: 1413
|
|
Re: 2007 qualifying for computer championship
« Reply #1 on: Nov 11th, 2005, 6:27pm » |
Quote Modify
|
I support this, it sounds like a nice way of preserving the mystery between bot developers, but allowing reasonable human experimentation.
|
|
IP Logged |
|
|
|
Ryan_Cable
Forum Guru
Arimaa player #951
Gender:
Posts: 138
|
|
Re: 2007 qualifying for computer championship
« Reply #2 on: Nov 12th, 2005, 4:03am » |
Quote Modify
|
I agree that this would be a better system, if only to assure humanity a shot at the bots in exactly their final configuration before the Challenge. However, for fairness sake, I think the three humans who are selected to defend the challenge should be prohibited from personally playing these final configuration bots. Otherwise they will be able to do what Fritzlein is describing only more so. I think reviewing the games other people play with the bots, swapping analysis in the forum, and playing the pre-freeze bot if the developer does voluntarily place it online should give them plenty of information and better approximates what preparation for a match with a human is like. In any event, I think that a bot developer would be at serious disadvantage by programming in secret and giving up all of the play testing opportunities offered by the gameroom.
Slightly off topic: I do think that a bot, especially one of the weaker bots, could gain an advantage in the Computer Championship if it had an opening book that incorporated one or more of the bot bashing strategies. The only hard part would be designing the conditionals to break out of the book if the target bot didn’t cooperate. It would be particularly advantageous if bot_bomb is going to be minimally changed from last year.
|
|
IP Logged |
|
|
|
Fritzlein
Forum Guru
Arimaa player #706
Gender:
Posts: 5928
|
|
Re: 2007 qualifying for computer championship
« Reply #3 on: Nov 12th, 2005, 10:21am » |
Quote Modify
|
Ryan,
It is good point that humans, when practicing against the final versions of the bots, might be able to discover a line which wins every time. However, there is a huge distinction in objective:
If I, as a bot developer, discover a line that beats your bot every time, that doesn't mean my bot is better, and it doesn't mean my bot deserves to win heads up against your bot in the Computer Championship.
If I, as a human defender of the Arimaa Challenge, discover a line that beats your bot every time, then your bot is no match for human intelligence, and therefore unworthy to win the challenge.
Admittedly, it is an open question how much time humans should get to probe for bugs and weaknesses. How good is a bot that wins the first ten games against the human World Champion, and then loses the next twenty? I would be very afraid of that bot the following year if its bugs got fixed, but I still say a machine that can't stand up under even two weeks of scrutiny deserves to lose right then. Once the challenge prize seems more within reach of the developers, they can focus more on preventing cheap tricks by the humans, just as Deep Blue was tuned to not be snookered by the anti-computer tactics of 1997.
(For what it's worth, I think Kasparov would have won against Deep Blue given two weeks of practice, but against Hydra today, two weeks would be insufficient. Training time is a buffer to allow human adaptability to come into play, but it doesn't protect us indefinitely.)
By the way (and maybe Fotland can clarify this) I'm pretty sure the winning program in the Gifu computer go championship did play differently depending on its opponent. Thus we may well currently have a computer champion in go which does not play as well against a human opponent as programs which finished lower in the standings. I don't think my concern here is idle speculation: It doesn't do Arimaa any good if bots start using cheap tricks against each other, especially if that results in the champion bot not being the strongest overall player against humans. We can't prevent that from happening, but we can discourage it.
|
|
IP Logged |
|
|
|
acheron
Forum Full Member
Arimaa player #1613
Gender:
Posts: 11
|
|
Re: 2007 qualifying for computer championship
« Reply #4 on: Nov 14th, 2005, 3:44pm » |
Quote Modify
|
I'd suggest changing it more than that.
Exposing your 'bot to online play beforehand is both a risk and an advantage. You gain the advantage of extensive testing to reveal flaws, and yet simultaneously expose any such flaws to outside analysis.
However, it is inequitable to treat computer players differently than human plays in this respect. No human is forced to make themselves available for forty days to have their own weaknesses probed. Instead, they enter the matches from a neutral starting point.
I'd remove the early showing requirement altogether. Let the developers choose whether or not to use the online facilities to refine their 'bot.
Let's face it, pre-generated paths for use against specific 'bots are in no way a test of intelligence, for either the 'bot or the humans. Is not the goal to create an environment where thinking is the key (adaptive particularly). If so, then all this scouting and pre-generation is purely counter to that goal.
|
|
IP Logged |
|
|
|
Fritzlein
Forum Guru
Arimaa player #706
Gender:
Posts: 5928
|
|
Re: 2007 qualifying for computer championship
« Reply #5 on: Nov 15th, 2005, 12:36pm » |
Quote Modify
|
on Nov 14th, 2005, 3:44pm, acheron wrote:| Let's face it, pre-generated paths for use against specific 'bots are in no way a test of intelligence, for either the 'bot or the humans. |
|
On the contrary, this is a test of intelligence in my book. If a bot can be beaten the same way every time, it clearly isn't intelligent. Conversely if a human can come up with a winning line against a bot, that shows intelligence somewhere in humanity, albeit more in the person who originates the line than in the person just clever enough to play it.
Quote:| Is not the goal to create an environment where thinking is the key (adaptive particularly). If so, then all this scouting and pre-generation is purely counter to that goal. |
|
I agree that thinking should definitely be the key in whatever format we choose. If bots become at all adaptive in the future, then a bot which which wins 90% at first will hardly be driven down to a 10% or 0% win rate by two weeks of exposure, since the bot can be learning at the same time. (Incidentally, by "final code", I mean only no further human modification. Self-modifying bots are fine.)
I think the general assumption is that humans are more adaptable than bots, and will continue to be more adaptable even than bots which can learn. Therefore we suppose that a shorter exposure favors machines while a longer exposure favors humans. The humans would absolutely not fear exposing themselves to bot challenges for forty days or for any period of time. So when we discuss longer or shorter practice before a match it is in the back of our minds how much we want to handicap the match in favor of bots or in favor of humans.
If we didn't think about that balance, and instead thought only about measuring adaptability and intelligence, then we could just rephrase the Arimaa Challenge to be "Put your bot online for a year with no intervention (only self-modification) and if no human can win 30 against it in a 40 game span, the bot wins." I actually would quite enjoy such a challenge structure, but in my opinion, that would handicap the bots even more than my proposal of two weeks' exposure of the final code and three mini-matches thereafter. Making a bot that plays Arimaa well and learns is probably much harder than just making a bot that plays well.
|
|
IP Logged |
|
|
|
acheron
Forum Full Member
Arimaa player #1613
Gender:
Posts: 11
|
|
Re: 2007 qualifying for computer championship
« Reply #6 on: Nov 15th, 2005, 1:19pm » |
Quote Modify
|
My real problem there is that now you've created an environment that is not reflective of the environment for a human player.
No human makes themselves available to such exhaustive study and repetitive testing.
Longer exposure only favors humans because it implies testing specifically for means of beating the one bot in question, not by the one human opponent, but by a multitude of bot bashers looking for weakness. Therein lies the problem.
Now the bright side of game records being listed does mean that it is somewhat possible to do this in reverse. It might be interesting, for example, to scour the records of the top few players and make particular note of the situations that led to their losses. Does player #2 have a tendency to under-value his Horses and suffer losses for trading them too lightly? Does player #3 struggle whenever the focal point is on the righthand side of the field instead of the lefthand where his thinking is more comfortable? Etc... An interesting concept, but unless you had a very large staff working on such a bot, impractical.
With the relatively low prize value, you're really talking hobby programmers. Such beforehand scouting then would fall upon his or their shoulders, as opposed to the hunter for 'bot weakness which has a larger crowd to draw upon.
My underlying point... don't set up a system that handicaps at all. Have the challenge decided by the performance at the actual event (even if this means extending the number of games played from 3 to 5 or 7) and not influenced by beforehand efforts of those not even directly involved.
|
|
IP Logged |
|
|
|
Janzert
Forum Guru
Arimaa player #247
Gender:
Posts: 1016
|
|
Re: 2007 qualifying for computer championship
« Reply #7 on: Nov 15th, 2005, 2:07pm » |
Quote Modify
|
on Nov 15th, 2005, 1:19pm, acheron wrote:| No human makes themselves available to such exhaustive study and repetitive testing. |
|
The lowest number of games played by a +2000 rated player is currently 299. Discounting omar_fast and bot_lightning, +1900 is 168 games. Minimum games +1800 is 99 games.
I think two weeks exposure without outside modification is a very short time and favors the bot.
Janzert
|
| « Last Edit: Nov 15th, 2005, 2:11pm by Janzert » |
IP Logged |
|
|
|
Fritzlein
Forum Guru
Arimaa player #706
Gender:
Posts: 5928
|
|
Re: 2007 qualifying for computer championship
« Reply #8 on: Nov 15th, 2005, 8:16pm » |
Quote Modify
|
on Nov 15th, 2005, 1:19pm, acheron wrote:Have the challenge decided by the performance at the actual event (even if this means extending the number of games played from 3 to 5 or 7) and not influenced by beforehand efforts of those not even directly involved.
|
|
It seems very fair and reasonable to simply hold an event and let the winner be the winner, without a bunch of preconditions. But you also hint at a big problem with this method, namely that a short match doesn't necessarily showcase intelligence, in particular not adaptability.
I believe that Kasparov would have beaten Deep Blue in a 20-game match instead of a 6-game match. He barely had any chance to adjust his strategies when he discovered his anti-computer ideas weren't working. Supposing that I am right, and Kasparov would have turned it around, then would it have been correct to say that Deep Blue was NOT the better chess player, even though it won the 6-game match?
For me, the ability to learn is built in to the definition of who is the best. I've beaten Omar in all eight of our games. What if next time we play, he surprises me with a new opening and beats me three times in a row with it? Would that make him better than me? I say it wouldn't, if I learned from those games and came back to beat him the next eight.
I guess what we're trying to avoid in the challenge rules is giving the prize to a bot that can win a short match but not a long one. Given that bots are less adaptable than humans, this is a reasonable probability. I hold that being beaten by a surprise you didn't figure out immediately shouldn't count if you can figure it out soon and win consistently after that. This certainly leaves open the question of how long is long enough, but it seems that the more you emphasize learning in your definition of being best, the longer you have to allow. For my preferences, a 20-game match would be long enough, even if the bot had no exposure other than the games themselves.
|
|
IP Logged |
|
|
|
omar
Forum Guru
Arimaa player #2
Gender:
Posts: 1003
|
|
Re: 2007 qualifying for computer championship
« Reply #9 on: Nov 16th, 2005, 1:36am » |
Quote Modify
|
Thanks for bring up this topic Karl. It is a very difficult and complex issue and one that I've thought about quite a bit. Yet I'm also not quite satisfied with the current solution and open to suggestions for improving it.
First of all, lets step back and think about why we need any qualifying games at all. Consider what happened in the DB vs GK match. The DB team had complete access to all the games GK had ever played while they developed DB in secrecy and it never played any public games until the day of the match. That I think is totally unfair. When two players face each other in a match they should have equal access to each others historical games records for preperation. Also the game record of both players should contain some minimum number of "serious" games; or both should contain nothing. Otherwise the situation is unfair to the player that has a publicly available historical game record, but has none or very limited such information about the opponent. Therefore bots need to play some minimum number of "serious" games so that there is a publicly available game records for opponents to review when they prepare to play against it match games.
Now consider the situation of the 2002 Kramnik vs Fritz match. Kramnik was given an exact copy of Fritz that he would play against in the match one month in advance so he could prepare for the event. Although this has been considered fair, I personally think that it is unfair to the bot. I don't think it is fair to have such complete access to a bot.
What I would really perfer is to have the bots play the "serious" games in a natural way throughout the year to establish a game record comprable to that of top human players. Then there would not be a need to impose any qualifying restrictions on them. But as we have seen the bots being developed usually are not available to play against throughout the year. Thus the need for imposing some qualifying restrictions. Perhaps someday when there are many Arimaa bots and the number of bots that can enter the championship tournament is limited with rating being a selection criteria then the bots will more naturally play "serious" games to qualify for the tournament and there will not be a need for any other qualifying requirements. However, the current situation does not support this. Thus the need for imposing some qualifying conditions.
I actually am not too concerned about the bots being changed till the last minute. I think it is very fair to allow the bot developers to continue changing the bots as much as they want. In fact the bots can even be changed between the match games as long as it is done in an automated way and does not require manual intervention from the bot developer. The model that I use for the bots is that of a program distributed to millions of people. The program developer will not be able to hand tweak the program to customize it to each end user, but he can ask the end user to run a program that automates the customization. Also there is no way one can prevent a bot developer from having the bot play differently during the freeze period than it does during the challenge match. The only thing that can be done is to discourage such practice. A freeze period would not prevent it. Also keep in mind that whenever a bot is changed the developer does not really know if the change actually improved the bot or made it worse. This gets to be a bigger and bigger problem as the bots improve to the point where hundreds of games are needed to determine if a change actually improved the bot. So I am not at all concerned about the bots being changed. What I am concerned about is that the bot developers be honest and present their current best bot as they would if they were trying to maximize it's ratings.
I initially had required the bots to play some minimum number of games to qualify; with a portion of those games being against humans. However, we know that some bots are played against more than others. So it could be possible that a bot does not meet the requirement just because humans did not play it; not because it was not available. So I changed the requirement to being available to play against; which is much more in the control of a bot developer.
Now 40 days might seem as you mentioned a long time for opponents to figure out exactly how to exploit a weakness. However, I think it is balanced out by allowing the bots to be changed during this time and up to the last minute. If an exploit is found a developer can try to fix it. They can even postpone fixing it to the last minute, but then they won't know if the change really fixed it especially if it is a strategic exploit and not just a simple bug in the code.
I think a solution which completely eliminate the need for qualifying conditions indirectly through other requirements which encourage the bots to play more regularly and try to maximize their ratings would be the best way to improve this.
|
|
IP Logged |
|
|
|
Fritzlein
Forum Guru
Arimaa player #706
Gender:
Posts: 5928
|
|
Re: 2007 qualifying for computer championship
« Reply #10 on: Nov 16th, 2005, 6:20pm » |
Quote Modify
|
on Nov 16th, 2005, 1:36am, omar wrote:| Also there is no way one can prevent a bot developer from having the bot play differently during the freeze period than it does during the challenge match. The only thing that can be done is to discourage such practice. A freeze period would not prevent it. |
|
Really? How would the bot know whether it was playing a challenge game or a pactice game? If the bot bases its decision of how to play on the date, you could alter the system clock or something to prevent that. I'm obviously not very savvy with this kind of thing, but I'm surprised that you don't think you could set it up somehow.
Quote:| I think a solution which completely eliminate the need for qualifying conditions indirectly through other requirements which encourage the bots to play more regularly and try to maximize their ratings would be the best way to improve this. |
|
Ensuring an adequate number of games for each bot at full strength is what I was trying to get at with a code freeze, so I'm curious to hear more about the problems with that idea. If the enviroment in which the submitted bots operated would somehow ensure that they had to play each game to win, because any game against another bot might be for the computer championship, and any game against a human might be part of the challenge, then you would have complete control over the amount of exposure you think is fair.
By the way, I'm still sort of intrigued by the notion of putting up bots on the server and declaring open season. I know this is a radical departure from the challenge match structure, but it seems to have an intuitive appeal to me.
* developer submits code
* bot is made available on the server
* if no human can do X within the next Y months, humanity acknowledges defeat
X could be any individual compiling a +20 score (wins minus losses) or whatever seems fair. If humans were brave enough, they could choose a faster time control than 2 minutes per move, but they wouldn't have the option of going slower than 2 minutes per move.
The developers could get in line for a turn at the challenge. Whenever a bot gets shot down, its developer would have to go to the back of the line, and the next contending bot could be put in place. The challenge could become a perpetual thing, a way of life for the Arimaa community. You can be sure humans would be falling over themselves to be the one to shoot down whichever contender was currently in place.
I dunno, it just seems kind of fun, but maybe it wouldn't work in practice. Or even if it would work, it would seem to the developers we are just moving the goalposts yet again.
Does anyone else think this would be an interesting idea?
|
|
IP Logged |
|
|
|
nbarriga
Forum Guru
Almost retired Bot Developer
Gender:
Posts: 119
|
|
Re: 2007 qualifying for computer championship
« Reply #11 on: Nov 16th, 2005, 6:41pm » |
Quote Modify
|
on Nov 16th, 2005, 6:20pm, Fritzlein wrote:
By the way, I'm still sort of intrigued by the notion of putting up bots on the server and declaring open season. I know this is a radical departure from the challenge match structure, but it seems to have an intuitive appeal to me.
* developer submits code
* bot is made available on the server
* if no human can do X within the next Y months, humanity acknowledges defeat
X could be any individual compiling a +20 score (wins minus losses) or whatever seems fair. If humans were brave enough, they could choose a faster time control than 2 minutes per move, but they wouldn't have the option of going slower than 2 minutes per move.
The developers could get in line for a turn at the challenge. Whenever a bot gets shot down, its developer would have to go to the back of the line, and the next contending bot could be put in place. The challenge could become a perpetual thing, a way of life for the Arimaa community. You can be sure humans would be falling over themselves to be the one to shoot down whichever contender was currently in place.
I dunno, it just seems kind of fun, but maybe it wouldn't work in practice. Or even if it would work, it would seem to the developers we are just moving the goalposts yet again.
Does anyone else think this would be an interesting idea?
|
|
I really like the idea, i think that having a system like this will encourage developpers to work during all the year. And you could set up a parallel bot ranking with the "highest number of [days|matches] undefeated while competing for the challenge".
|
|
IP Logged |
|
|
|
nbarriga
Forum Guru
Almost retired Bot Developer
Gender:
Posts: 119
|
|
Re: 2007 qualifying for computer championship
« Reply #12 on: Nov 16th, 2005, 6:41pm » |
Quote Modify
|
on Nov 16th, 2005, 6:20pm, Fritzlein wrote:
By the way, I'm still sort of intrigued by the notion of putting up bots on the server and declaring open season. I know this is a radical departure from the challenge match structure, but it seems to have an intuitive appeal to me.
* developer submits code
* bot is made available on the server
* if no human can do X within the next Y months, humanity acknowledges defeat
X could be any individual compiling a +20 score (wins minus losses) or whatever seems fair. If humans were brave enough, they could choose a faster time control than 2 minutes per move, but they wouldn't have the option of going slower than 2 minutes per move.
The developers could get in line for a turn at the challenge. Whenever a bot gets shot down, its developer would have to go to the back of the line, and the next contending bot could be put in place. The challenge could become a perpetual thing, a way of life for the Arimaa community. You can be sure humans would be falling over themselves to be the one to shoot down whichever contender was currently in place.
I dunno, it just seems kind of fun, but maybe it wouldn't work in practice. Or even if it would work, it would seem to the developers we are just moving the goalposts yet again.
Does anyone else think this would be an interesting idea?
|
|
I really like the idea, i think that having a system like this will encourage developpers to work during all the year. And you could set up a parallel bot ranking with the "highest number of [days|matches] undefeated while competing for the challenge".
|
|
IP Logged |
|
|
|
Janzert
Forum Guru
Arimaa player #247
Gender:
Posts: 1016
|
|
Re: 2007 qualifying for computer championship
« Reply #13 on: Nov 16th, 2005, 11:50pm » |
Quote Modify
|
on Nov 16th, 2005, 6:20pm, Fritzlein wrote:| By the way, I'm still sort of intrigued by the notion of putting up bots on the server and declaring open season. I know this is a radical departure from the challenge match structure, but it seems to have an intuitive appeal to me. |
|
While I really like this idea from the pure "let's best try and determine if bots or humans are better at arimaa" point of view, I think it has a number of shortcomings when looking at ancillary concerns.
Probably top most of these would be, for spectators there is no annual "Grand match" focal point. Probably resulting in greater difficulty getting more sponsors as well.
Janzert
|
|
IP Logged |
|
|
|
Fritzlein
Forum Guru
Arimaa player #706
Gender:
Posts: 5928
|
|
Re: 2007 qualifying for computer championship
« Reply #14 on: Nov 18th, 2005, 9:39am » |
Quote Modify
|
on Nov 16th, 2005, 11:50pm, Janzert wrote:| for spectators there is no annual "Grand match" focal point. Probably resulting in greater difficulty getting more sponsors as well. |
|
Good point. I wonder if bot developers would participate if it were set up as a side contest, independent of the Arimaa Challenge. We have the Player of the Month contest to encourage humans to play each other. In parallel there could be a regular Bot Contender prize to encourage developers to work on their bots year-round. The prize money could be split between the developer and the first player to garner a +20 score, depending on how long the bot survived. (There I go giving away Omar's money...) We could have a hall of fame so that defenders of human supremacy would gain eternal glory.
I'm just brainstorming here. The Arimaa server is already a way cool place to play. It seems that bot development has cooled off a bit from last year, but it is probably cyclical and will pick up again without any structural changes.
|
|
IP Logged |
|
|
|
|
Home
Help
Search
Members
Login
Register
Reply
Notify of replies
Send Topic
Print