Author |
Topic: Will the 2010 Computer Championship be open? (Read 13759 times) |
|
Fritzlein
Forum Guru
Arimaa player #706
Gender: 
Posts: 5928
|
 |
Re: Will the 2010 Computer Championship be open?
« Reply #45 on: Aug 20th, 2009, 6:19am » |
 Quote  Modify
|
on Aug 19th, 2009, 7:54pm, Janzert wrote:| I think it is also a favor to the other developers that qualify as well as the all the spectators. |
|
Hmm, I wonder if all the developers that have to jump through hoops while watching the previous champion rest on his laurels would agree that exempting the previous champion is a favor to them too.
In the chess world, FIDE tried holding knock-out tournaments, but wanted to make sure the previous champion would be involved in the climactic final match instead of getting eliminated early, so they seeded the previous champion directly into the finals. I assure you that the rank and file did not view it as a favor to themselves that the previous champion had an easier path to victory, regardless of whether it preserved the glory of the event. It's a different situation, true, but the sentiment might be similar for Arimaa.
|
|
 IP Logged |
|
|
|
tize
Forum Guru
Arimaa player #3121
Gender:
Posts: 118
|
|
Re: Will the 2010 Computer Championship be open?
« Reply #46 on: Aug 20th, 2009, 7:49am » |
Quote Modify
|
Why not just give the developer of the reigning champ a unique decision, either he register the old bot unmodified and get the bot automatic qualified, or he register a new bot and have to qualify it like everybody else.
If the developer doesn't register any bot, Omar can register the champ. (auto qualified)
|
|
IP Logged |
|
|
|
omar
Forum Guru
Arimaa player #2
Gender:
Posts: 1003
|
|
Re: Will the 2010 Computer Championship be open?
« Reply #47 on: Aug 20th, 2009, 7:55am » |
Quote Modify
|
Regarding the cheating issue. I don't think we should worry about it too much. As Arimabuff mentioned there isn't much incentive to do it. In fact, the developer would probably just lose the entry fee in the finals and risk being banned from future tournaments. If the hardware used by the developer was much, much faster than the hardware of the tournament server, it could also appear as if the developer was cheating. I may need to publish the hardware that will be used on the tournament server before the qualifier and there might need to be a clause in the rules to says that during the qualifier the developers should not use a system that is more than twice as fast as the tournament server. If there is suspicion of cheating, I can run the bot from the tournament server against the benchmark bots to see if it performs similar to how it did during the qualifying phase. The tournament director can look at this performance difference while taking into consideration the hardware used by the developer compared to the tournament server and decide if there has been cheating. There should be a clause in the rules that if the tournament director determines a developer cheated during the qualifying phase then the developer loses the entry fee, is not entitled to any awards or prizes and will be banned from entering a bot in the following years tournament.
Regarding automatic entry in the final for the previous champion. The motivation for this rule seems to be to guarantee continuity from one year to the next of the champion being in the finals. Continuity is desirable, but it is possible the bot developer just doesn't have the time to improve the bot and does not enter a new bot at all; we've seen this happen already. In such a case I will enter the previous years champion in the qualifier. The probability of the best bot from one year not making it to the finals the very next year is low enough that I don't think we need a special rule to guarantee it. Although it could happen in the future when there are say 30 really good bots competing for the 8 positions in the finals. There could be big changes in position from one year to the next. But having such a rule opens up the door for the Deep Blue syndrome. That is, the developer of the champion bot can enter a bot into the finals which has not played any public games at all. This bot could be very different than the champion bot. I think this is very unfair to all the other bots in the final; especially if the bot that got automatic entry looks at their public games to decided how to play against them and they have no access to it's games. GnoBot already does look at previous games and we have to assume that this may become the norm in the future. So for this reason I would not want to have a special exemption for the developer of the champion bot. Also as has been mentioned the previous years champion bot will be available during development season, so I am sure the other bots will have played against it, especially the bot that will go on to become the new champion.
Karl I didn't quite get this:
Quote:
Thus if the qualifying bot has played the benchmark bot twenty times with that color since the starting date, count only the fourth and fifth games.
|
|
Did you mean to say "... has played the benchmark bot five times with that color ..."; because I thought you suggested a maximum of 5 games with each color.
I am afraid that counting only the last two games with each color against the benchmark bots will not provide enough discrimination. It will be a problem if the really good bots all get a perfect score. Then we have to rely on the tie breaker which counts the total number of qualifying games played. This measure is also not forgiving to allow for multiple plays to get a better result. In addition the really good bots might only need to play two games with each color to get the perfect score. This would require going into the second tie breaker which is not based on the merits of the bot. If this happens I don't think we would be very happy with how the qualifying phase went.
My reason for wanting to use a performance rating was to allow discrimination between the bots based only on merit. Rather than assigning fixed ratings to the benchmark bots, we could use a weighted point system for wins and losses. For examples a win against the first lowest benchmark bot is worth 1 point, against the second is worth 2 points and so on. If there are 8 benchmark bots then a win against the best benchmark bot is worth 8 points. Losses count for equivalent negative points; or to encourage more playing the negative points for a loss could be half the positive points for a win. To allow some leeway to improve the bot during the qualifying phase and try for a better score, one loss with each color against each benchmark bot will not be counted. As tize mentioned there is no need to require a minimum number of games. So there is only a cap on the maximum number of games which can be played against each benchmark bot for each color. This should allow the bots to compete with each other to maximize their score during the qualifying phase and provide sufficient discrimination. In the event of tied scores a blitz tie break game can be played between the tied bots. The bot which achieved the score first gets to pick the color in the tie break game.
|
| « Last Edit: Aug 22nd, 2009, 7:18am by omar » |
IP Logged |
|
|
|
99of9
Forum Guru
Gnobby's creator (player #314)
 
Gender:
Posts: 1413
|
|
Re: Will the 2010 Computer Championship be open?
« Reply #48 on: Aug 27th, 2009, 10:01pm » |
Quote Modify
|
I'm with Janzert.
However, one thing I'd like to ensure is that any bot allowed in by a special route still plays a minimum number of games to "show" as much of itself as the other bots show. [EDIT: I now read that Omar also spoke of this. But in response to Omar, it is still possible to have both a requirement to play qualifiers, and a free pass even if you fail to qualify.]
Omar's argument that the 2009CC is unlikely to miss out in 2010 is not a reason for or against having a rule. The question is, if or when it DOES miss out, do we want to give it special consideration? I do.
I'm with ArimaaBuff on the cheating issue. There's hardly any incentive, and this is the kind of situation where the trustworthiness of the Arimaa community comes in handy. If we were really to get paranoid on cheating, monitoring players in the WC to ensure they were not running bots would be more important IMO.
|
| « Last Edit: Aug 27th, 2009, 10:14pm by 99of9 » |
IP Logged |
|
|
|
Fritzlein
Forum Guru
Arimaa player #706
Gender:
Posts: 5928
|
|
Re: Will the 2010 Computer Championship be open?
« Reply #49 on: Sep 17th, 2009, 3:04pm » |
Quote Modify
|
I apologize for dropping the ball on getting new rules written up. I have now at least copied the 2009 rules here, and have begun editing.
I suggest the following eight benchmark bots:
* ArimaaScoreP1
* Loc2007P1
* Arimaazilla
* Aamira2006P2
* Bomb2005Fast
* OpFor2009Fast
* Gnobot2009Blitz
* Clueless2009Blitz
My selection criteria were:
1) We want a range of benchmarks from the worst bots to the best, so that we provide discrimination throughout the spectrum. It would be silly to do a good job of seeding the top four qualifying bots but a terrible job of deciding which bot should qualify into the top eight compared to the bot that doesn't qualify in ninth place. Conversely, we don't want all the discrimination at the lower end while all the best qualifying bots tie for perfect score at the top end, because that wouldn't give us a good seeding.
2) In case a developer has an advantage in beating previous versions of his own bot, there should be no more than one benchmark bot from any developer.
3) There should be a variety of time controls. For the lower four benchmark bots the qualifying bot has a time advantage, but for the upper four benchmark bots, the qualifying bot has no time advantage.
Any further suggestions? Next year we can adjust the stable of benchmark bots if this collection appears somehow lacking during 2010 qualification.
on Aug 20th, 2009, 7:55am, omar wrote:| I may need to publish the hardware that will be used on the tournament server before the qualifier |
|
If you are going to have a rule that says the qualifying bots can't run on super-fast hardware, you at least need to specify the hardware limit, if not the exact hardware that will be used for the tournament.
Quote:| Continuity is desirable, but it is possible the bot developer just doesn't have the time to improve the bot and does not enter a new bot at all; we've seen this happen already. In such a case I will enter the previous years champion in the qualifier. |
|
I like this solution because it guarantees continuity without opening the can of worms of a developer having two bots. If a developer wants to enter a wholly different bot, perhaps one that is much weaker than his champion of last year, then that new bot will have to prove itself by qualifying. If the champion developer doesn't have the time or inclination to qualify, then Omar can run his previous bot through the qualifying in order to get a reasonable seeding and insure that the quality of the tournament is at least not lower than the previous year.
If the situation ever arises that either the old version or the new version of a championship bot fails to qualify in the top eight after being entered, then I say boot it out and give the eighth spot to a more worthy contender. If so many new and/or improved bots have surpassed the old champion, there is no possible issue with the quality of the tournament being low.
Quote:Karl I didn't quite get this:
Did you mean to say "... has played the benchmark bot five times with that color ..."; because I thought you suggested a maximum of 5 games with each color. |
|
Yes, I suggested a maximum of five games against each benchmark bot during the qualifying. Can you set up some hard limit on the benchmark bots so that they don't play extra? In the absence of an automatically-enforced limit, I thought we should have a rule to cover what should be done when a developer plays a sixth and seventh game. One could say that the bot in question is automatically disqualified, but that seems a wee bit harsh. I thought it would be nicer to instead ignore any games beyond the maximum (i.e. any games beyond the fifth). Thus no matter how many extra games a qualifying bot plays against a benchmark bot with a certain color, we ignore all extras and just look at the last two legitimate games, i.e. the fourth and fifth. Does this make sense?
Quote:| I am afraid that counting only the last two games with each color against the benchmark bots will not provide enough discrimination. It will be a problem if the really good bots all get a perfect score. Then we have to rely on the tie breaker which counts the total number of qualifying games played. This measure is also not forgiving to allow for multiple plays to get a better result. In addition the really good bots might only need to play two games with each color to get the perfect score. |
|
I would be astonished if any bot entered for the 2010 championship could rack up 32 wins against this lineup with no losses. To have a reasonable shot of running the table, the new bot would need to be at least 250 rating points better than the best bot of last year. Do you really expect so much improvement? My expectation is that several bots at the top will be able to max out their score at 32, but also that the number of tries needed to get those wins will provide sufficient discrimination.
Quote:| My reason for wanting to use a performance rating was to allow discrimination between the bots based only on merit. Rather than assigning fixed ratings to the benchmark bots, we could use a weighted point system for wins and losses. For examples a win against the first lowest benchmark bot is worth 1 point, against the second is worth 2 points and so on. If there are 8 benchmark bots then a win against the best benchmark bot is worth 8 points. Losses count for equivalent negative points; or to encourage more playing the negative points for a loss could be half the positive points for a win. |
|
Introducing an arbitrary number of points for wins and losses against certain bots seems like a complication that not only has no benefit, it actually makes things worse as well as more complicated. How do you know that your point system is fair? You could make it more lucrative to play against one bot than against another, in which case careful selection of opponents will be part of the skill in getting a high seed for a qualifying bot. I strongly recommend not having any scoring system in which self-selection of opponents can provide any advantage. Selection of opponents is what wrecks the ordinary ratings, and it seems like a great danger to you ad-hoc scoring as well.
Besides, if you think this year's bots are strong enough to win and win against the old bots without losing, how does it help matters to make the scoring system more complicated? A perfect score is a perfect score in any system. The only way you can add discrimination is by making more of the games count. That's fine by me: if you want to make a total of 80 games count in the scoring instead of having 32 games count in the scoring, then let's do that for greater discrimination. Let's not count the last two games against each benchmark bot with each color; instead let's count all five games.
I repeat, however, that an arbitrary scoring system with self-selection of opponents does not help you in any way. Instead we should define the opponents exactly (however we decide) and count one point per win among the defined games.
Quote:| In the event of tied scores a blitz tie break game can be played between the tied bots. The bot which achieved the score first gets to pick the color in the tie break game. |
|
A playoff blitz game is a reasonable alternative to break ties instead of using time of completion.
|
|
IP Logged |
|
|
|
jdb
Forum Guru
Arimaa player #214
Gender:
Posts: 682
|
|
Re: Will the 2010 Computer Championship be open?
« Reply #50 on: Sep 17th, 2009, 5:07pm » |
Quote Modify
|
Quote:| Yes, I suggested a maximum of five games against each benchmark bot during the qualifying. Can you set up some hard limit on the benchmark bots so that they don't play extra? In the absence of an automatically-enforced limit, I thought we should have a rule to cover what should be done when a developer plays a sixth and seventh game. One could say that the bot in question is automatically disqualified, but that seems a wee bit harsh. I thought it would be nicer to instead ignore any games beyond the maximum (i.e. any games beyond the fifth). Thus no matter how many extra games a qualifying bot plays against a benchmark bot with a certain color, we ignore all extras and just look at the last two legitimate games, i.e. the fourth and fifth. Does this make sense? |
|
Bots entering the tournament are most likely under development, and changing over time. If there is a limit on the number of games against each bot, the developer will wait until the last minute to play the games. If the rules only count the last n games against each bot, then there is much less incentive to hold off playing games.
The last 4 bots on the list are not fixed performance bots. Thats OK, but it is something to be aware of.
|
|
IP Logged |
|
|
|
Arimabuff
Forum Guru
Arimaa player #2764
Gender:
Posts: 589
|
|
Re: Will the 2010 Computer Championship be open?
« Reply #51 on: Sep 17th, 2009, 9:18pm » |
Quote Modify
|
on Sep 17th, 2009, 3:04pm, Fritzlein wrote:| ...Yes, I suggested a maximum of five games against each benchmark bot during the qualifying... |
|
I think that any rule that limits the number of games played by a bot is a very bad idea. The whole idea behind this site is to promote the game of Arimaa and multiply the possibilities to play it. NOT the other way around. Also, we must encourage the bot developers to… well… develop their bots as much as possible and we will not do that if we limit the number of test games that they are allowed to play.
|
|
IP Logged |
|
|
|
Arimabuff
Forum Guru
Arimaa player #2764
Gender:
Posts: 589
|
|
Re: Will the 2010 Computer Championship be open?
« Reply #52 on: Sep 17th, 2009, 9:22pm » |
Quote Modify
|
on Sep 17th, 2009, 5:07pm, jdb wrote:
Bots entering the tournament are most likely under development, and changing over time. If there is a limit on the number of games against each bot, the developer will wait until the last minute to play the games. If the rules only count the last n games against each bot, then there is much less incentive to hold off playing games.
The last 4 bots on the list are not fixed performance bots. Thats OK, but it is something to be aware of.
|
|
Point well taken, we should encourage the evolving bot vis a vis the fixed one. Also a developer who tests his bot against human players is more likely to do a better job than one who doesn't.
|
|
IP Logged |
|
|
|
Fritzlein
Forum Guru
Arimaa player #706
Gender:
Posts: 5928
|
|
Re: Will the 2010 Computer Championship be open?
« Reply #53 on: Sep 19th, 2009, 10:37am » |
Quote Modify
|
on Sep 17th, 2009, 5:07pm, jdb wrote:| The last 4 bots on the list are not fixed performance bots. Thats OK, but it is something to be aware of. |
|
Using only fixed-performance bots would be ideal for fairness, but unfortunately fixed-performance bots by their nature give a time handicap. Therefore the strongest available bots are not fixed-performance. Since we are worried about discrimination between the best bots, we need the top of the ladder to be as strong as possible. Therefore the top benchmark bots should be variable-performance.
In the middle of the ladder, using Bomb2005P2 instead of Bomb2005Fast would probably be better in terms of having the rungs of the ladder equally spaced, and would be fairer by virtue of being fixed-performance. The drawback of this substitution would be that BombP2 can be beaten by imitating past bot-bashes, whereas BombFast is more variable and therefore presumably less prone to duplicating whole games.
How do people feel about whether it should be BombP2 or BombFast among the benchmark bots?
Quote:| Bots entering the tournament are most likely under development, and changing over time. If there is a limit on the number of games against each bot, the developer will wait until the last minute to play the games. If the rules only count the last n games against each bot, then there is much less incentive to hold off playing games. |
|
If developers hold off playing their games until the last minute, it creates the possible problem of the server not having the resources to play all the qualifying games at once. If we mandate playing eighty games (rather than merely allowing eighty games), the problem of congestion would be exacerbated.
This consideration makes me want to revive my proposed second tiebreak, namely earliest completion of qualifying, to give at least a small incentive not to wait until the last minute. Also, on further consideration, I don't like having a blitz playoff game as a tiebreak because it would be a logistical hassle to arrange the game in advance of the programs being ported to the server. I will provisionally change the rule Wiki to use time as the second tiebreaker, but of course I'm not trying to end the discussion thereby.
on Sep 17th, 2009, 9:18pm, Arimabuff wrote:| I think that any rule that limits the number of games played by a bot is a very bad idea. The whole idea behind this site is to promote the game of Arimaa and multiply the possibilities to play it. NOT the other way around. Also, we must encourage the bot developers to… well… develop their bots as much as possible and we will not do that if we limit the number of test games that they are allowed to play. |
|
I agree with your general point about encouraging development and play, but I still think we had better cap the number of qualifying games. Removing the limit on the number of qualifying games rewards persistence more than further development. A developer whose bot beats Bomb only 10% of the time with a month to go before the tournament can get a two-game winning streak by playing about a hundred games against Bomb. This would be an easier and more reliable way to improve the qualifying score than by adding last-minute features. I don't think we want to encourage developers to set up their bots to play incessantly until getting a win streak of the right length.
If we define the stable of benchmark bots now, developers have over three months of unlimited practice to test new features and fix bugs. Also, even during qualifying, developers can test and practice as much as they want against any bots other than the eight benchmarks. We are by no means stifling developers from playing, testing, and developing. We are merely trying to come up with a reasonable way to measure performance. Allowing unlimited games interferes with the performance measurement.
I think the policy of allowing five qualifying games and counting the last two is a good compromise between accommodating last-minute bugfixes on one hand and keeping the scoring related to actual bot ability on the other hand.
|
| « Last Edit: Sep 19th, 2009, 10:40am by Fritzlein » |
IP Logged |
|
|
|
jdb
Forum Guru
Arimaa player #214
Gender:
Posts: 682
|
|
Re: Will the 2010 Computer Championship be open?
« Reply #54 on: Sep 19th, 2009, 1:00pm » |
Quote Modify
|
Quote:| For bots which tie on qualifying score, the first tiebreaker is score minus the number of qualifying games played. Thus the best possible tiebreak score is zero, which represents winning all qualifying games and not playing any excess games. The worst possible tiebreak score is -80, which represents playing the maximum number of games and winning none. The second tiebreak is earliest date of completion, i.e. the bot whose last qualifying game was played first. |
|
I hope someone can clarify this for me. If two bots are equal in score, the first tiebreaker can simply be the number of qualifying games played. Or am I missing something?
I like the second tiebreak. No need for a playoff game.
|
|
IP Logged |
|
|
|
Fritzlein
Forum Guru
Arimaa player #706
Gender:
Posts: 5928
|
|
Re: Will the 2010 Computer Championship be open?
« Reply #55 on: Sep 19th, 2009, 2:29pm » |
Quote Modify
|
on Sep 19th, 2009, 1:00pm, jdb wrote:| I hope someone can clarify this for me. If two bots are equal in score, the first tiebreaker can simply be the number of qualifying games played. Or am I missing something? |
|
That's what I intended for the first tiebreak. Did I not present it clearly?
|
|
IP Logged |
|
|
|
Fritzlein
Forum Guru
Arimaa player #706
Gender:
Posts: 5928
|
|
Re: Will the 2010 Computer Championship be open?
« Reply #56 on: Sep 23rd, 2009, 8:58am » |
Quote Modify
|
While chatting with JDB today, it became clear that we must not split winning streaks against benchmark bots into streaks as Gold and streaks as Silver. The discrimination provided by qualifying is vastly greater if the longest winning streak that counts is four rather than two. That's just how the math works out. So we are going to have to require that qualifying bots play the benchmark bots with alternating colors.
I am sad advocate an alternating color requirement, because I know that will make it impossible for Omar to create a leader board with straight SQL queries. Instead Omar will have to query for all the games of a qualifying bot versus a benchmark bot in date order and step through those games looking for valid winning streaks. This is a hassle, I know, but will be extremely valuable in adding discrimination between qualifying bots without adding extra games.
JDB and I also discussed a maximum number of games allowed per benchmark bot. I wanted a low limit to stop the server from being pounded by games, and to stop a weaker bot from passing a stronger bot by pure persistence. JDB wanted no limit, to allow for open-ended development. In the end, I think we were both satisfied with allowing twenty games per benchmark bot. A developer who wants to play more than the 160-game maximum within the qualifying month is probably not focusing much on developing anyway. Hopefully this limit is low enough that the server doesn't get swamped.
What do you think?
|
|
IP Logged |
|
|
|
jdb
Forum Guru
Arimaa player #214
Gender:
Posts: 682
|
|
Re: Will the 2010 Computer Championship be open?
« Reply #57 on: Sep 23rd, 2009, 9:13am » |
Quote Modify
|
on Sep 23rd, 2009, 8:58am, Fritzlein wrote:
I am sad advocate an alternating color requirement, because I know that will make it impossible for Omar to create a leader board with straight SQL queries. Instead Omar will have to query for all the games of a qualifying bot versus a benchmark bot in date order and step through those games looking for valid winning streaks. This is a hassle, I know, but will be extremely valuable in adding discrimination between qualifying bots without adding extra games.
What do you think? |
|
It might be easier to enforce the alternating colour requirement on the other end. Enforce the alternation of colour before the game is played, instead of checking for it after.
|
|
IP Logged |
|
|
|
Fritzlein
Forum Guru
Arimaa player #706
Gender:
Posts: 5928
|
|
Re: Will the 2010 Computer Championship be open?
« Reply #58 on: Sep 23rd, 2009, 9:59am » |
Quote Modify
|
on Aug 11th, 2009, 9:44am, RonWeasley wrote:I would like to see specific language in the rules about handling server failures or degradations. My policy of continuing the game at the point of failure was not unanimously supported and it put quite a burden on Omar. If the rules are amended to call for a restart, for example, whenever a server failure is detected, the TD would not have to be consulted each time and tournament management would be more tractable.
Also think about handling server failures in the qualifying games and the effects of restarts or continuations on players. A simple restart policy might be the most effective in dealing with scheduling. |
|
I have put tentative language about handling server failures in the proposed rules. My proposal for dealing with issues during qualifying is that everything counts no matter what happens. This may sound arbitrary and harsh, but it would be a nightmare for Omar to have to deal with issues for a whole month like he has to deal with for the two-week tournament.
For in-tournament issues, I think we should agree on a list of issues that cause manadatory halt/restart, in order to take the judgment of the Tournament Director out of the equation. Should we also list any issues that don't qualify for a halt/restart?
|
|
IP Logged |
|
|
|
Fritzlein
Forum Guru
Arimaa player #706
Gender:
Posts: 5928
|
|
Re: Will the 2010 Computer Championship be open?
« Reply #59 on: Oct 15th, 2009, 6:01am » |
Quote Modify
|
If Fotland doesn't return to active development this year, we may have the issue of whether to enter Bomb in the qualification process or not. Bomb is not the defending champion any more, so we needn't worry about continuity in that sense.
My feeling is that if there are seven or fewer other entrants, it is fine to add Bomb to fill out the field, but if there are eight or more active developers, then we shouldn't enter Bomb on behalf of an inactive developer. Even if Bomb would still be stronger than the bot in the eighth spot, I think it is less important to have the strongest field possible, and more important to include a bot with a future in front of it. Let's not have deadwood taking up a slot.
How do other people feel about this rule clarification? That is to say, we will automatically enter an unimproved reigning champion, but will not automatically enter an unimproved bot otherwise?
|
|
IP Logged |
|
|
|
|
Home
Help
Search
Members
Login
Register
Reply
Notify of replies
Send Topic
Print