Author |
Topic: Will the 2010 Computer Championship be open? (Read 13758 times) |
|
omar
Forum Guru
Arimaa player #2
Gender: 
Posts: 1003
|
 |
Re: Will the 2010 Computer Championship be open?
« Reply #75 on: Nov 5th, 2009, 9:38am » |
 Quote  Modify
|
on Oct 28th, 2009, 5:04pm, aaaa wrote:| Wouldn't it be better if the first tiebreaker only applied to all the qualifying games up to the last one that contributed to the primary score? Otherwise, there could be an incentive for one to stop early. |
|
Suppose against a particular benchmark bot, two qualifying bots A and B have a record like this:
A: 10111
B: 101110
Assuming that A and B have an identical record against all other bots and thus tie on qualifying score. aaaa is saying that since the first tie break is the score minus the number of qualifying games played bot B would get a lower first tie break score because it attempted one more game than bot A. This would provide an incentive to not continue playing after reaching a fairly good winning streak. I tend to agree that only the games up to the last game of the longest winning streak should be subtracted from the score in computing the first tie break. So the first tie break could probably be stated as: the score minus the total number of losses prior to the longest winning streak.
|
|
 IP Logged |
|
|
|
omar
Forum Guru
Arimaa player #2
Gender:
Posts: 1003
|
|
Re: Will the 2010 Computer Championship be open?
« Reply #76 on: Nov 5th, 2009, 9:57am » |
Quote Modify
|
on Oct 31st, 2009, 3:15pm, jdb wrote:
This still seems a little harsh to me.
What is gained if the bot plays an unrated game against one of the ladder bots? Clearly, the game doesn't count in the score, so what is the benefit of playing the unrated game? I can easily setup bot_clueless_jr and play against them anyway. Its most likely just a developer mistake setting things up. I don't understand the need to penalize for a mistake.
|
|
I guess the intent was that during the qualifying period practice games played as unrated games should not be allowed against the benchmark bots. If unrated games were allowed it might allow a qualifying bot to play unrated games to adjust to the benchmark bot before playing the rated games which count. But as jdb mentioned this could still be done using a second bot that is not registered. So I am for removing the restriction that unrated games can't be played against the benchmark bot during the qualifying phase unless someone can provide a strong case for keeping it.
|
|
IP Logged |
|
|
|
Fritzlein
Forum Guru
Arimaa player #706
Gender:
Posts: 5928
|
|
Re: Will the 2010 Computer Championship be open?
« Reply #77 on: Nov 5th, 2009, 1:08pm » |
Quote Modify
|
on Oct 28th, 2009, 5:04pm, aaaa wrote:| Otherwise, there could be an incentive for one to stop early. |
|
If there is no incentive to stop early, I expect we will see most or all qualifying bots play all twenty games against all eight benchmark bots, except when they achieve a four-game winning streak. If we are going to use up the server resources by having all the games played anyway, we might as well change the scoring to count all the games, so as to get better discrimination between bots.
The longer I mull over the "winning streak" scoring, the less I like it. It throws away lots of information in the cause of encouraging last-minute development. But what our scoring scheme will probably do is encourage developers to play more games instead of developing more, because there is greater payoff in the scoring for persistence than there is for tinkering to gain minor improvements. Furthermore, with 160 games per developer, I expect contention for who gets to play the benchmark bots, particularly at the last minute.
I think we would get a better seeding by requiring exactly four games against each benchmark bot, i.e. 32 games per developer, and counting them all. To get around jdb's concern of every developer waiting until the last minute to play the qualifying games, we could give a slight bonus for playing early, say 0.2% per day before the deadline. I have no qualms having seeding favor the best bot as of January 1 rather than the best bot as of January 31 if that means we will not have to clog up the server with qualifying games.
|
|
IP Logged |
|
|
|
omar
Forum Guru
Arimaa player #2
Gender:
Posts: 1003
|
|
Re: Will the 2010 Computer Championship be open?
« Reply #78 on: Nov 10th, 2009, 7:11pm » |
Quote Modify
|
I updated to rules to incorporate the suggestions given by aaaa and jdb.
Quote:| Furthermore, with 160 games per developer, I expect contention for who gets to play the benchmark bots, particularly at the last minute. |
|
I think the first four bots should be pretty easy for the qualifying bots.
Quote:| To get around jdb's concern of every developer waiting until the last minute to play the qualifying games, we could give a slight bonus for playing early, say 0.2% per day before the deadline. |
|
I think this year we might be too late to make more changes to the rules. So lets hold off until next year for this.
|
|
IP Logged |
|
|
|
aaaa
Forum Guru
Arimaa player #958
Posts: 768
|
|
Re: Will the 2010 Computer Championship be open?
« Reply #79 on: Nov 11th, 2009, 7:09am » |
Quote Modify
|
The second tiebreaker also needs rewriting as the way it is phrased would imply that it could only be used to differentiate between bots if either has played all possible qualifying games, including possible excessive ones. In the same spirit behind the change of the first tiebreaker, I would therefore turn "earliest date of completion" into "earliest date of reaching one's final score" or words to that effect.
|
|
IP Logged |
|
|
|
omar
Forum Guru
Arimaa player #2
Gender:
Posts: 1003
|
|
Re: Will the 2010 Computer Championship be open?
« Reply #80 on: Nov 12th, 2009, 5:59pm » |
Quote Modify
|
Thanks. I change it to read:
The second tie-break is earliest date of completion, i.e. the bot whose last game which counts towards the score was played first.
|
|
IP Logged |
|
|
|
aaaa
Forum Guru
Arimaa player #958
Posts: 768
|
|
Re: Will the 2010 Computer Championship be open?
« Reply #81 on: Nov 14th, 2009, 7:18am » |
Quote Modify
|
If you want to leave absolutely no room for interpretation, replace "played" by "finished".
|
|
IP Logged |
|
|
|
omar
Forum Guru
Arimaa player #2
Gender:
Posts: 1003
|
|
Re: Will the 2010 Computer Championship be open?
« Reply #82 on: Nov 24th, 2009, 1:13pm » |
Quote Modify
|
Changed it; thanks.
|
|
IP Logged |
|
|
|
omar
Forum Guru
Arimaa player #2
Gender:
Posts: 1003
|
|
Re: Will the 2010 Computer Championship be open?
« Reply #83 on: Dec 19th, 2009, 8:00pm » |
Quote Modify
|
Here is the bot qualifying program. Please check and let me know if you see any problems.
For now I've set only four bots as the qualifying bots; quad, clueless, marwin and OpFor.
Also for testing I've set the the time frame for the games to last two years.
A break down of all the games that were used in computing the score is listed along with how the score and tie breaks were computed.
The requirement for playing with alternating colors is observed as follows. The first game against a benchmark bot sets the color and subsequent games must be with alternating colors. Otherwise it breaks the streak even if the game was won; and if the game was lost it adds to the loses if it occurred prior to the the longest streak.
http://arimaa.com/arimaa/wcc/2010/qual.htm
|
|
IP Logged |
|
|
|
Janzert
Forum Guru
Arimaa player #247
Gender:
Posts: 1016
|
|
Re: Will the 2010 Computer Championship be open?
« Reply #84 on: Dec 19th, 2009, 8:49pm » |
Quote Modify
|
A few small comments.
I expected a streak to include the first game of the streak, i.e. one win is a streak of one.
I think losses fits the tense of the page better than loses if it's not a pain to change.
Janzert
|
|
IP Logged |
|
|
|
tize
Forum Guru
Arimaa player #3121
Gender:
Posts: 118
|
|
Re: Will the 2010 Computer Championship be open?
« Reply #85 on: Dec 20th, 2009, 5:59am » |
Quote Modify
|
Code:
bot_OpFor
...
bot_Loc2007P1
s L 1220896327 82968
s L 1220930920 82985
s W 1220934422 82990
streak is 0, loses is 2, total score is 0, total loses is 2, ...
...
bot_clueless
...
bot_Bomb2005Fast
s L 1203799256 71259
s W 1233711016 96104
s L 1233749824 96136
s W 1233775674 96157
g W 1233784044 96164
g W 1233787675 96169
s W 1233795579 96188
s W 1233955791 96400
g W 1234270647 96839
s W 1234899619 97612
s L 1235056316 97833
g W 1235063963 97840
s W 1235068572 97846
streak is 2, loses is 2, total score is 2, total loses is 2, ...
|
|
OpFor should have a streak one, and clueless should have a streak of 3. Other than that it looks good.
|
| « Last Edit: Dec 20th, 2009, 6:07am by tize » |
IP Logged |
|
|
|
omar
Forum Guru
Arimaa player #2
Gender:
Posts: 1003
|
|
Re: Will the 2010 Computer Championship be open?
« Reply #86 on: Dec 20th, 2009, 8:14am » |
Quote Modify
|
on Dec 19th, 2009, 8:49pm, Janzert wrote:
I expected a streak to include the first game of the streak, i.e. one win is a streak of one.
|
|
Yes, a single win counts as a streak of one; for example search the page for: 91061.
Quote:
I think losses fits the tense of the page better than loses if it's not a pain to change.
|
|
Thanks. Changed it.
|
|
IP Logged |
|
|
|
omar
Forum Guru
Arimaa player #2
Gender:
Posts: 1003
|
|
Re: Will the 2010 Computer Championship be open?
« Reply #87 on: Dec 20th, 2009, 8:36am » |
Quote Modify
|
on Dec 20th, 2009, 5:59am, tize wrote: Code:
bot_OpFor
...
bot_Loc2007P1
s L 1220896327 82968
s L 1220930920 82985
s W 1220934422 82990
streak is 0, loses is 2, total score is 0, total loses is 2, ...
...
bot_clueless
...
bot_Bomb2005Fast
s L 1203799256 71259
s W 1233711016 96104
s L 1233749824 96136
s W 1233775674 96157
g W 1233784044 96164
g W 1233787675 96169
s W 1233795579 96188
s W 1233955791 96400
g W 1234270647 96839
s W 1234899619 97612
s L 1235056316 97833
g W 1235063963 97840
s W 1235068572 97846
streak is 2, loses is 2, total score is 2, total loses is 2, ...
|
|
OpFor should have a streak one, and clueless should have a streak of 3. Other than that it looks good. |
|
For OpFor game 82990 does not count because of the alternating color requirement. Game 82968 was played as silver so the next game which can increase the streak must be played as gold.
For clueless the streak gets broken several times because of the alternating color requirement. Game 71259 sets the initial color to silver, so the next game which can increase the streak must be played as gold. Game 96164 is the next game which increases the streak count to 1 since it was played with gold. But the very next game, 96169, breaks the streak since it was not played as silver even though the game was won. Because the streak gets broken several times the longest streak is two games.
|
| « Last Edit: Dec 20th, 2009, 8:38am by omar » |
IP Logged |
|
|
|
jdb
Forum Guru
Arimaa player #214
Gender:
Posts: 682
|
|
Re: Will the 2010 Computer Championship be open?
« Reply #88 on: Dec 20th, 2009, 8:46am » |
Quote Modify
|
Clueless has a winning streak of 5 against bot_arimaascorep1. It should only count as 4 points towards the total score.
|
|
IP Logged |
|
|
|
Sconibulus
Forum Guru
Arimaa player #4633
Gender:
Posts: 116
|
|
Re: Will the 2010 Computer Championship be open?
« Reply #89 on: Dec 20th, 2009, 9:31am » |
Quote Modify
|
Omar, as I read it, clueless should still have a streak of 3 due to this segment.
s W 1233955791 96400
g W 1234270647 96839
s W 1234899619 97612
I think the reason that it doesn't is that when a streak is broken due to colour-fault, it doesn't count that game as a start of a new streak. Basically it seems to count that win as a loss, rather than a streak-breaking win.
|
|
IP Logged |
|
|
|
|
Home
Help
Search
Members
Login
Register
Reply
Notify of replies
Send Topic
Print