Author |
Topic: 2007 qualifying for computer championship (Read 4011 times) |
|
omar
Forum Guru
Arimaa player #2
Gender: 
Posts: 1003
|
 |
Re: 2007 qualifying for computer championship
« Reply #30 on: Nov 22nd, 2005, 10:39am » |
 Quote  Modify
|
Karl, you are absolutely right, the qualifying games really are trying to kill two birds with one stone and although the two events are seperate they are in some ways intangled. It reallys helps to recognise this fact explicitly.
Perhaps it would be better to seperate the qualifying games for the two events, but as Toby mentioned it raises some issues about fairness to the bots. So we must be very careful in how we collect the record of games against humans.
Consider the following:
After the computer championship tournament is over the two bots which finished with the best results compete to determine which will play in the challenge match.
The two bots will be kept online for humans to play against for two weeks. Any human player that wants to play against the bots must play only two games against each bot once with each color. So many humans can play against the bots during the two weeks, but each can only play two games and must play two games against both bots. The human players selected for the challenge match must not play the bots during this period and can only observe and analyse the games. Remember the human players for the challenge match are already determined before the computer championship tournament begins. The bot which has a better record after the two weeks with ties broken using number of moves in the games will go on to play in the challenge match.
|
|
 IP Logged |
|
|
|
Fritzlein
Forum Guru
Arimaa player #706
Gender:
Posts: 5928
|
|
Re: 2007 qualifying for computer championship
« Reply #31 on: Nov 22nd, 2005, 2:10pm » |
Quote Modify
|
That's a fantastic idea, Omar. Intuitively it addresses the fairness issue to forbid the actual defenders of the Challenge match from playing practice games against the bot contenders, and to limit everyone else to two each.
I don't think bot developers can complain too much about it. In particular, if a human outside of the top three can come up with a winning line during qualifying that the bot falls for verbatim in the Challenge, then that bot deserves to lose the Challenge.
This format would be extremely exciting if bots ever advanced to the point that only three humans could beat the best bot consistently. Imagine the fever pitch of interest in the Challenge match if some bot were winning against all its human opponents during the two-week qualifying phase!
At the same time, I think the proposed structure is fair to humanity, because there is a large enough pool of strong players nowadays to try out all the basic strategies, which gives the defenders some idea of what to expect no matter how those games turn out.
But what I like best about your proposal, Omar, is that it makes the qualifying more appropriate to the event. The bot we would most want to participate in the Challenge isn't the best anti-bot bot, it is the best anti-human bot.
If humans continue to be way ahead of bots, I can imagine developers will simply give up on the Challenge, and instead focus on getting their bot to beat other bots. Someone who does a good job of that could theoretically win the Computer Championship with a bot that is easier for humans to beat than the losers of the Computer Championship.
If the top two bots from the Computer Championship have a run-off qualifier for the Challenge based on anti-human play, it will reward a bot developer who focuses also on beating humans, and not purely on beating other bots. And regardless of developers' intentions, your qualifying rules would give us a better chance of selecting the bot most skilled in anti-human play.
|
|
IP Logged |
|
|
|
99of9
Forum Guru
Gnobby's creator (player #314)
 
Gender:
Posts: 1413
|
|
Re: 2007 qualifying for computer championship
« Reply #32 on: Nov 23rd, 2005, 12:05am » |
Quote Modify
|
on Nov 22nd, 2005, 10:39am, omar wrote:| The bot which has a better record after the two weeks with ties broken using number of moves in the games will go on to play in the challenge match. |
|
An interesting twist. Unfortunately it opens the possibility of humans trying to manipulate the bot qualification. If the best bot was way ahead of the second best bot, humans could throw games against the second best bot in order to ensure their champions did not have to face the number 1 bot. Obviously I don't think this is all that likely, but it would be nice if it were excluded.
I still think the simplest temporal order would be:
1) Choose human participants (submit to ICGA only)
2) Submit bots
3) Human participants revealed
4) Humans allowed to play any bots they want for time T
5) Bot tournament.
6) Winner of bot tournament is challenger
7) Challenge
|
|
IP Logged |
|
|
|
Fritzlein
Forum Guru
Arimaa player #706
Gender:
Posts: 5928
|
|
Re: 2007 qualifying for computer championship
« Reply #33 on: Nov 24th, 2005, 1:07pm » |
Quote Modify
|
on Nov 23rd, 2005, 12:05am, 99of9 wrote:| An interesting twist. Unfortunately it opens the possibility of humans trying to manipulate the bot qualification. If the best bot was way ahead of the second best bot, humans could throw games against the second best bot in order to ensure their champions did not have to face the number 1 bot. Obviously I don't think this is all that likely, but it would be nice if it were excluded. |
|
This is a valid objection, but I so much like the idea of using bot vs. human games in qualifying for a bot vs. human event that I want to take a close look at whether this objection can be overcome.
First, I do think that at least developers should be excluded from playing their own bots, because clearly developers have an incentive to throw games against their own bot. So the two developers and the three defenders are out. I concede that at present this makes the pool of potential human players smaller, and that a smaller pool of humans makes this qualifying scheme more open to manipulation. However, the small numbers could be meliorated somewhat by putting the two contending bots in the lobby and encouraging everyone who has beaten Arimaazilla to play them. Also there would be more participation if the time control were set to something reasonably brisk, say 45 seconds per move.
Second, Omar could write into the rules a provision that all four games will be disregarded for anyone who has, in Omar's opinion, purposely played weakly in any game. But I admit that the subjective nature of poor play means this rule could be enforced only in extreme cases.
The third and most important protection, however, comes from the motivation of human players. How will humanity (apart from the five already ruled out) behave, and why?
Right now, I imagine most people will want to see the strongest available bot play for the title, even if they are rooting for the bot to lose. Nobody thinks even the best bot has a chance to win at present, so bot-haters will want to see the best bot in the Challenge so it can be decisvely crushed. For different reasons, the people who would want the bot to win the Challenge will certainly want the best bot there. And then there are the majority of honest people who want the top bot playing for the prize just because that's the way it should be.
But let's fast forward a few years to a worst case scenario. Suppose a new, super-strong bot comes out and crushes the competition in the bot tournament. Suppose also the bot has no game history against humans, and no losses against bots, so nobody knows if it even has any weaknesses. Suddenly humanity would be afraid of losing the Challenge.
A few people (probably a small percentage, but let's say they exist) are motivated by not wanting humanity to fail, and are also unscrupulous enough to try to throw the qualifying in favor of the weaker bot. However, in order to succeed, they must be able to beat the strongest bot, and indeed be able to do so in their first two tries ever against that bot. I say that if there exists a cadre of people outside of the defenders who beat the bot without practice, presumably that bot was scant threat to humanity anyway.
One could say that even if humanity has a 100% chance of winning the Challenge, it is still unfair to the bot developer whose bot lost in qualifying to an inferior bot. It is true that this would be unfair. But if humanity has no chance of losing the Challenge, I don't see where humans get the motivation to manipulate bot qualifying. What's the point? So either the motivation or the ability to be dishonest will always be lacking. It seems that this minimizes the potential problem to the point that we might reasonably ignore it.
But I do see it is a balancing act. If bots qualify for the Challenge by playing humans rather than by playing other bots, it has some very nice positives, but I see how negatives can creep in as well.
Quote:
I still think the simplest temporal order would be:
1) Choose human participants (submit to ICGA only)
2) Submit bots
3) Human participants revealed
4) Humans allowed to play any bots they want for time T
5) Bot tournament.
6) Winner of bot tournament is challenger
7) Challenge |
|
If Omar decides to keep it that the winner of the bot tourney automatically becomes the challenger, then this order of events appears to be very sensible. Depending on the length of time T, it protects the interests of both the bots developers and the Challenge defenders.
|
|
IP Logged |
|
|
|
omar
Forum Guru
Arimaa player #2
Gender:
Posts: 1003
|
|
Re: 2007 qualifying for computer championship
« Reply #34 on: Nov 28th, 2005, 4:09pm » |
Quote Modify
|
on Nov 23rd, 2005, 12:05am, 99of9 wrote:
An interesting twist. Unfortunately it opens the possibility of humans trying to manipulate the bot qualification. If the best bot was way ahead of the second best bot, humans could throw games against the second best bot in order to ensure their champions did not have to face the number 1 bot. Obviously I don't think this is all that likely, but it would be nice if it were excluded.
|
|
Good point Toby. I had a feeling someone would bring this up. This possiblility occured to me, but I didn't think it would be too much of a problem. For your statement to hold, we have to make the assumption that somehow people already know which bot is stronger against humans. I would argue that people playing the two bots would not already know this, if the bots have not given much of a historical record against humans. The new format does not require any record against humans prior to the computer championship.
|
|
IP Logged |
|
|
|
omar
Forum Guru
Arimaa player #2
Gender:
Posts: 1003
|
|
Re: 2007 qualifying for computer championship
« Reply #35 on: Nov 28th, 2005, 4:20pm » |
Quote Modify
|
on Nov 24th, 2005, 1:07pm, Fritzlein wrote:
A few people (probably a small percentage, but let's say they exist) are motivated by not wanting humanity to fail, and are also unscrupulous enough to try to throw the qualifying in favor of the weaker bot. However, in order to succeed, they must be able to beat the strongest bot, and indeed be able to do so in their first two tries ever against that bot. I say that if there exists a cadre of people outside of the defenders who beat the bot without practice, presumably that bot was scant threat to humanity anyway.
|
|
Yes, I think this is also a very strong argument as to why we need not worry about everyone conspiring to throw games against the weaker bot to get it into the challenge.
However, consider a senerio where both bots have not lost any games, but because people purposely lost faster to the weaker bot (assuming they somehow already knew which was weaker), the weaker bot gets selected based on the number of moves rule.
Perhaps to further eliminate that possibility we could add an exception to the rules that if both bots have not been defeated then the bot which won the computer championship goes on to play in the challenge match.
|
| « Last Edit: Nov 28th, 2005, 4:36pm by omar » |
IP Logged |
|
|
|
omar
Forum Guru
Arimaa player #2
Gender:
Posts: 1003
|
|
Re: 2007 qualifying for computer championship
« Reply #36 on: Nov 28th, 2005, 4:33pm » |
Quote Modify
|
on Nov 23rd, 2005, 12:05am, 99of9 wrote:
I still think the simplest temporal order would be:
1) Choose human participants (submit to ICGA only)
2) Submit bots
3) Human participants revealed
4) Humans allowed to play any bots they want for time T
5) Bot tournament.
6) Winner of bot tournament is challenger
7) Challenge |
|
We would still have the problem that in step 4 we don't know if the bot was really playing to it's full potential. The bot has no steak in those games. So as I mentioned earlier, it would be possible to look at the time to decide how well to play. I also plan to add an 'Event' field to the game info that is available to the bots, so even that could be used to decide how to play. The bots need to have a steak in the games they play against the humans when establishing their record against humans. Ratings are usually enough motivation, but tournaments and contests make it even more stronger.
|
|
IP Logged |
|
|
|
Fritzlein
Forum Guru
Arimaa player #706
Gender:
Posts: 5928
|
|
Re: 2007 qualifying for computer championship
« Reply #37 on: Nov 29th, 2005, 5:35pm » |
Quote Modify
|
on Nov 28th, 2005, 4:20pm, omar wrote:However, consider a senerio where both bots have not lost any games, but because people purposely lost faster to the weaker bot (assuming they somehow already knew which was weaker), the weaker bot gets selected based on the number of moves rule.
Perhaps to further eliminate that possibility we could add an exception to the rules that if both bots have not been defeated then the bot which won the computer championship goes on to play in the challenge match.
|
|
Or, more simply, if the top two bots end up tied on won-loss record, then the tie is broken in favor of the bot that won the Computer Championship, and game length doesn't enter into it ever. With that tiebreak in effect, you would actually have to beat a bot to conspire against it, rather than merely losing faster to the other bot.
|
| « Last Edit: Nov 29th, 2005, 5:45pm by Fritzlein » |
IP Logged |
|
|
|
Fritzlein
Forum Guru
Arimaa player #706
Gender:
Posts: 5928
|
|
Re: 2007 qualifying for computer championship
« Reply #38 on: Nov 29th, 2005, 5:44pm » |
Quote Modify
|
on Nov 28th, 2005, 4:33pm, omar wrote:| The bots need to have a stake in the games they play against the humans when establishing their record against humans. |
|
I hadn't quite understood this part of your proposal before, Omar. By having the bot vs. human games be part of a runoff qualifying process, you help ensure that the bots will try their hardest in the very games where we need them to be playing at full strength.
When you add it to all the other benefits of qualifying for the Challenge on the basis of games versus humans, the benefit of enouraging full-strength play ices the cake. Even though I 'm still slightly worried about the possibility of conspiracy against a particular bot, I think the proposed runoff system is much better than what we have now.
|
| « Last Edit: Dec 12th, 2005, 5:53pm by Fritzlein » |
IP Logged |
|
|
|
|
Home
Help
Search
Members
Login
Register
Reply
Notify of replies
Send Topic
Print