Author |
Topic: 2007 Computer Championship (Read 2900 times) |
|
Fritzlein
Forum Guru
Arimaa player #706
Gender: 
Posts: 5928
|
 |
Re: 2007 Computer Championship
« Reply #30 on: Jan 20th, 2007, 11:51am » |
 Quote  Modify
|
on Jan 20th, 2007, 5:37am, DorianGaray wrote:Looks like we're not making much progress, are we?  |
|
Indeed. In some sense Fairy/Zombie represents progress, but if an unaltered Bomb can win three straight years...
|
|
 IP Logged |
|
|
|
IdahoEv
Forum Guru
Arimaa player #1753
Gender:
Posts: 405
|
|
Re: 2007 Computer Championship
« Reply #31 on: Jan 22nd, 2007, 2:59am » |
Quote Modify
|
I note that the WCC page shows that Zombie and Bomb have played a finals game. However, I also note that they played the same sides (Zombie on gold and Bomb on silver) as they played in their round 3 game. Given that they have not played with Bomb on gold, was this the correct pairing?
Incidentally, the play was completely identical up through 14b. On 14w Zombie deviated by one step from the previous game, which cascaded into a different endgame.
note: I'll report on rounds 7 and 8 in the morning, I haven't been around on the weekend.
|
| « Last Edit: Jan 22nd, 2007, 3:05am by IdahoEv » |
IP Logged |
|
|
|
Fritzlein
Forum Guru
Arimaa player #706
Gender:
Posts: 5928
|
|
Re: 2007 Computer Championship
« Reply #32 on: Jan 22nd, 2007, 9:04am » |
Quote Modify
|
on Jan 22nd, 2007, 2:59am, IdahoEv wrote:| I note that the WCC page shows that Zombie and Bomb have played a finals game. However, I also note that they played the same sides (Zombie on gold and Bomb on silver) as they played in their round 3 game. Given that they have not played with Bomb on gold, was this the correct pairing? |
|
The color assignment is based on the total number of times each player has played each color in the tournament. If you add across all earlier rounds, Zombie had three gold and three silver, while Bomb had four gold and two silver. Therefore Zombie was due gold and Bomb was due silver. If you believe that playing gold is an advantage, then it would be an unfair advantage to Bomb to get to play gold five times in seven games. If you believe that playing silver is an advantage, then it would be an unfair disadvantage to Bomb to have to play gold five times in seven games.
On the other hand, since gold and silver are so far statistically indistinguishable in winning percentage, evening out the color advantage may seem secondary to avoiding repeated games. It would have been lame for Zombie to lose twice in exactly the same way. I was very happy that Zombie deviated, and happier that Zombie's deviation was actually an improvement over its previous line. It was just bad luck that it provoked an improved line of play from Bomb, and led to an even quicker loss for Zombie.
I personally think it is up to bot developers to prevent their bots from repeating losses within a tournament. If a bot is dumb enough to lose twice in exactly the same way, it deserves both losses. I believe that in computer chess tournaments, bots handle this via their opening book, i.e. if a program loses a game, it will choose a different line within its opening book for the remainder of the tournament.
|
| « Last Edit: Jan 22nd, 2007, 11:25am by Fritzlein » |
IP Logged |
|
|
|
RonWeasley
Forum Guru
Harry's friend (Arimaa player #441)
Gender:
Posts: 882
|
|
Re: 2007 Computer Championship
« Reply #33 on: Jan 22nd, 2007, 9:36am » |
Quote Modify
|
Reading the tournament rules closely I found color assignment constraints only in the tie breaker section. While there may be justification for either color in this last game (see IdahoEv and Fritzlein above), the tie breaker rules favor the colors that were used. Since this game has been played, I rule that the result stands.
Perhaps color assignment rules could be more explicit next year.
Tournament Director
|
|
IP Logged |
|
|
|
IdahoEv
Forum Guru
Arimaa player #1753
Gender:
Posts: 405
|
|
Re: 2007 Computer Championship
« Reply #34 on: Jan 22nd, 2007, 4:10pm » |
Quote Modify
|
Reading the rules, I agree with the TD. Given that most of the bots are deterministic, it might be more interesting to have the pairing rules avoid repeating identical matchups, but Fritzlein makes a good point that avoiding repetition of losses should probably be on the head of the bots rather than the tournament rules.
|
|
IP Logged |
|
|
|
IdahoEv
Forum Guru
Arimaa player #1753
Gender:
Posts: 405
|
|
Re: 2007 Computer Championship
« Reply #35 on: Jan 22nd, 2007, 4:39pm » |
Quote Modify
|
The tournament is concluded, with Bomb defeating and eliminating Clueless in round 7, then defeating and eliminating Zombie in round 8. Final standings are below. As per the tournament rules, ties are broken by pre-tournament ratings.
Congratulations to all the developers for an excellent tournament.
Final Standings:
| Place | Bot | Record | Defeated | Defeated by | | Winner | Bomb | 7-0 | GnoBot,Faerie,Zombie (x2), Aamira, Clueless (x2) | | 2nd | Zombie | 4-3 | Occam, Aamira, Faerie (x2) | Bomb (x2), Clueless | | 3rd | Clueless | 3-3 | GnoBot,Loc, Zombie | Faerie, Bomb (x2) | | 4th | Faerie | 3-3 | Clueless, Aamira, Loc | Bomb, Zombie (x2) | | 5th | Aamira | 2-3 | Loc, Occam | Zombie, Faerie, Bomb | | 6th | Loc | 1-3 | Occam | Clueless, Aamira, Faerie | | 7th | Occam | 1-3 | GnoBot | Zombie, Loc, Aamira | | 8th | GnoBot | 0-3 | | Bomb, Clueless, Occam |
Round 7/8 Games report: (click the heading to open game viewer)
Clueless vs. Bomb*
Clueless opened with an EH attack and a threat on a dog that Bomb had left exposed in the initial opening. However, Clueless failed to make god on the dog threat. In the fight over the dog, Bomb first threatened gold's horse, then camel, leveraging these threats to keep Clueless always five or six steps away from either capturing the dog or taking control of the trap threatened by the E-H attack. Eventually, Bomb used this to create simultaneous material threats on three of gold's pieces (20w): camel, horse, and rabbit, while clueless had no effective threats on the board. This led to a rapid cascading loss of several of gold's pieces over the next few turns. Clueless attempted to respond with a goal race, but did not prevail.
Zombie vs. Bomb*
This was an identical repeat of Game 46124 up through turn 14b; zombie opening with an EMH attack that traded Zombie's H for Bomb's D and giving up a camel hostage. In response, Zombie attempted to threaten a dog and develop goal threats, but with a decentralized elephant and slightly down on material it simply didn't have sufficient tactical control over the board to make good on the threats.
|
| « Last Edit: Jan 22nd, 2007, 10:43pm by IdahoEv » |
IP Logged |
|
|
|
Fritzlein
Forum Guru
Arimaa player #706
Gender:
Posts: 5928
|
|
Re: 2007 Computer Championship
« Reply #36 on: Jan 25th, 2007, 3:43pm » |
Quote Modify
|
I'm pleased that floating triple-elimination worked this time around. The bottom four bots were each eliminated with losses to three different bots. That takes away a potential criticism that a bot got unluckily paired with opponents that just happened to be able to exploit a quirk.
Yes, a round-robin would be the most fair to the lower bots, but given that the total number of games that can be played is limited by server rental time, it is better to spend the extra games on repeat pairings of the top bots than on extra mismatches which are unlikely to provide additional information.
Of course Bomb laid to rest any doubt about its dominance, beating five of the seven other bots, and beating both #2 and #3 twice. IdahoEv wins my bet with him about whether Bomb could go undefeated.
Due to the new qualifying phase against humans, it is important that the top two bots be accurately selected, not just the top one, and I think the format did a pretty good job of that too. Zombie clearly has bragging rights over Faerie after winning both head-to-head games. It was only close between Clueless and Zombie for second place due to Zombie's inability to avoid repetition in an otherwise excellent position. Quirks matter, but it was a quirk. It reminds me of 2005 when Bomb eked out Clueless only on tiebreakers, but Bomb was clearly the better bot.
I'm glad triple-elimination worked well here, because I still hold out hope that Omar will switch the format of the World Championship to be triple-elimination as well. 
|
| « Last Edit: Jan 25th, 2007, 3:46pm by Fritzlein » |
IP Logged |
|
|
|
IdahoEv
Forum Guru
Arimaa player #1753
Gender:
Posts: 405
|
|
Re: 2007 Computer Championship
« Reply #37 on: Jan 26th, 2007, 6:18pm » |
Quote Modify
|
on Jan 25th, 2007, 3:43pm, Fritzlein wrote:| Yes, a round-robin would be the most fair to the lower bots, but given that the total number of games that can be played is limited by server rental time, it is better to spend the extra games on repeat pairings of the top bots than on extra mismatches which are unlikely to provide additional information. |
|
Just to provide a contrary view:
Given that the CC does not have the scheduling conflict problems that the WC does, and computers can be scheduled to play anytime or even started automatically by a script, I think a round robin is completely doable without increasing server rental costs.
It wouldn't cost more because the current system wastes a lot of time: it prevents scheduling of a subsequent round until the previous one has finished and the TD has reviewed the results of the pairing algorithm. If you do a round-robin, all pairings have to be completed so there's no need for that review and the games can be automated. That saves the day-plus that currently went by in between rounds. (No complaint, Omar - I couldn't possibly have kept up with the reporting without that day!)
Anyway, consider: Write a script that automatically starts the next game in the queue when the last one finishes. If anything goes wrong with a game, the TD can just rule to have a new game inserted into or appended onto the queue.
If the average bot game takes 4 hours (more than generous based on recent results), then a full round-robin of 8 bots completes in 168 hours, or 7 days. (8 choose 2 is 28 pairings making 56 games to get both color combinations. 56*4=168.) If I am not mistaken, Omar is already renting the two tournament servers for quite a bit longer than that.
You could schedule them at exact times every 4 hours, which would take the full 7 days but would allow people to know in advance when a game was if they wanted to watch it live, or you could start new games immediately on completion of the others which would shorten the total required time considerably.
For 6 bots, it's 30 games = 90 hours = 3.75 days
For 8 bots, 56 games = 168 hours = 7 days
For 10 bots, 90 games = 270 hours = 11.25 days
For 12 bots, 132 games = 396 hours = 16.5 days
That's based on 4 hour games, which is generous. This year's tournament would have been 7 bots except for the inclusion of Occam. It seems unlikely we'd go above 10 bots next year.
|
| « Last Edit: Jan 26th, 2007, 6:22pm by IdahoEv » |
IP Logged |
|
|
|
|
Home
Help
Search
Members
Login
Register
Reply
Notify of replies
Send Topic
Print