Author |
Topic: Will the 2010 Computer Championship be open? (Read 13765 times) |
|
99of9
Forum Guru
Gnobby's creator (player #314)
 
Gender: 
Posts: 1413
|
 |
Re: Will the 2010 Computer Championship be open?
« Reply #120 on: Jan 20th, 2010, 2:12pm » |
 Quote  Modify
|
I'll reply properly later, but just a note that Gnobot2009Blitz timeouts could cause even more issues than with marwin... it is happening against gnobot too.
|
|
 IP Logged |
|
|
|
omar
Forum Guru
Arimaa player #2
Gender:
Posts: 1003
|
|
Re: Will the 2010 Computer Championship be open?
« Reply #121 on: Jan 20th, 2010, 2:40pm » |
Quote Modify
|
on Jan 15th, 2010, 2:23am, tize wrote:
These four losses shouldn't count in the tie-breaker score as they are all played after the winning streak of 0. |
|
OK I changed it so that, losses do not count unless there was a non-zero winning streak.
|
|
IP Logged |
|
|
|
omar
Forum Guru
Arimaa player #2
Gender:
Posts: 1003
|
|
Re: Will the 2010 Computer Championship be open?
« Reply #122 on: Jan 20th, 2010, 2:52pm » |
Quote Modify
|
on Jan 20th, 2010, 2:12pm, 99of9 wrote:| I'll reply properly later, but just a note that Gnobot2009Blitz timeouts could cause even more issues than with marwin... it is happening against gnobot too. |
|
The system load goes up quite a bit when any version of GnoBot2009 runs. I think this may be causing the timeouts. I have each version capped at one instance, but different versions could get started at the same time.
Maybe we should not count games where a benchmark bot loses on time.
|
| « Last Edit: Jan 20th, 2010, 2:53pm by omar » |
IP Logged |
|
|
|
Fritzlein
Forum Guru
Arimaa player #706
Gender:
Posts: 5928
|
|
Re: Will the 2010 Computer Championship be open?
« Reply #123 on: Jan 20th, 2010, 3:18pm » |
Quote Modify
|
on Jan 20th, 2010, 2:52pm, omar wrote:The system load goes up quite a bit when any version of GnoBot2009 runs. I think this may be causing the timeouts. I have each version capped at one instance, but different versions could get started at the same time.
Maybe we should not count games where a benchmark bot loses on time. |
|
Maybe GnoBot2009 should not be a benchmark bot, and indeed should not be available for play by anyone. Probably the load is caused by GnoBot2009 starting multiple threads, and it is unacceptable for any single bot instance to use up more than one CPU. For the future you should ask developers to include a single-threaded mode, and if no single-threaded mode is included, not let the resource-hogging bot run on the main server, period.
In terms of having a fair qualifying procedure, it is a poor stopgap to discard games that end on time. It is unfair for the server to be under load that might make a bot perform worse in one game than in another. If GnoBot uses all four CPUs when it is the only bot running, but uses only one, two, or three when other bots are running, then its performance could vary drastically from game to game.
The best way to make qualifying as fair as possible is to prevent the server from ever being under high load. Admittedly, it does some violence to the rules to drop from eight benchmark bots down to seven, but that seems less unfair than counting some games and not others, and much less unfair than having such a high load on the system that we can't rely on relatively consistent performance from the benchmark bots.
Ron, if you are reading this, what do you think of removing GnoBot2009 from the pool of benchmark bots?
|
| « Last Edit: Jan 20th, 2010, 3:27pm by Fritzlein » |
IP Logged |
|
|
|
99of9
Forum Guru
Gnobby's creator (player #314)
Gender:
Posts: 1413
|
|
Re: Will the 2010 Computer Championship be open?
« Reply #124 on: Jan 20th, 2010, 3:58pm » |
Quote Modify
|
on Jan 20th, 2010, 8:25am, Fritzlein wrote:
Yes, I think book wins are less legitimate for this purpose because they are a less robust indicator of tournament performance. If someone comes up with a way to ace the qualifying, we want that to also be a way to ace the tournament.
|
|
I agree that this is the right comparison to make. But it works in reverse too. If a method works in the tournament, it should work to the same extent in qualifying. Since the only data we have about what works in a tournament is what happened last year, I'd say that the book did "work" to some degree.
on Jan 20th, 2010, 8:25am, Fritzlein wrote:
I think book wins within the computer tournament are entirely legitimate, and I'm very happy for GnoBot's book win over Bomb in 2009 for exactly the reasons you mention. Because of GnoBot, every developer should implement some kind of defense against losing the same way twice. But whatever defense is implemented by developers in 2010 can't be backported to old benchmark bots.
|
|
You are right about what should happen, but we know that this will not be the case for all bots (since bomb is re-entered), and don't yet know how well the other bots will respond to the challenge. Since we can't know until after each CC tourney, I suggest that next year's qualifier bots should roughly reflect this year's entrants.
on Jan 20th, 2010, 8:25am, Fritzlein wrote:
Furthermore, if everyone is like JDB in letting parallelism be the main line of defense, but Omar turns off parallel search for game room bots (as I think he must to conserve server resources), then opening books could be extremely effective against benchmark bots while being wholly ineffective in the tournament itself. That kind of disconnect would be a flaw in the qualifying procedure.
|
|
Has Omar turned off parallelism?
On the other hand, server load fluctuations are much bigger in the benchmarking than they are in the real tournament (as we've seen with Gnobot2009Blitz... it never timed out in the real tourney, so it can't be blamed entirely). This is making the book less effective than it would be if this were last year's tournament.
|
|
IP Logged |
|
|
|
Fritzlein
Forum Guru
Arimaa player #706
Gender:
Posts: 5928
|
|
Re: Will the 2010 Computer Championship be open?
« Reply #125 on: Jan 20th, 2010, 6:58pm » |
Quote Modify
|
on Jan 20th, 2010, 3:58pm, 99of9 wrote:| Since we can't know until after each CC tourney, I suggest that next year's qualifier bots should roughly reflect this year's entrants. |
|
That sounds eminently reasonable. As long as the benchmark bots approximate the previous year's tournament, I'm happy. My biggest concern on that regard is the presence/absence of parallelism.
Quote:| Has Omar turned off parallelism? |
|
I thought Omar had set the 2009 bots to run in single-threaded mode on the server, but then he posted about GnoBot2009 causing high load, which convinces me that he didn't disable parallelism.
My suggestion is that bots should never be run in multi-threaded mode on the main server, because even one such bot will interfere with the performance of all other bots that happen to be running. And we should be especially sure that there are no multi-threaded bots in the pool of benchmark bots, because the performance of such bots will vary wildly.
Quote:| server load fluctuations are much bigger in the benchmarking than they are in the real tournament |
|
The performance of single-threaded bots will definitely vary more on the main server than it varies on the dedicated tournament server, but as long as each single-threaded bot has a free CPU it can grab (i.e. as long as four or fewer such bots are running simultaneously) their performance should be reasonably stable. The variation will at least be less than for a multi-threaded bot which always tries to use all the available CPUs.
|
| « Last Edit: Jan 20th, 2010, 7:01pm by Fritzlein » |
IP Logged |
|
|
|
99of9
Forum Guru
Gnobby's creator (player #314)
Gender:
Posts: 1413
|
|
Re: Will the 2010 Computer Championship be open?
« Reply #126 on: Jan 20th, 2010, 7:19pm » |
Quote Modify
|
That puts us in a conundrum. We want qualifier bots to reflect the previous year's entrants (which are now on the whole parallel). But we don't want them to ever run in parallel?
I understand where you're coming from, so I'm not sure what is the best thing to do.
I suppose since we're only really aiming to select the top 8 entrants, we don't really need the qualifier bots to be operating at their top standard, just a consistent standard. So in that case your solution is right. Even with a 4:1 processor advantage, the 9th place bot is unlikely to get a streak of 4 against the previous year's winner.
Perhaps the number of test games should be cut from 20 to 10 per bot? That would make it even more challenging to get a string of 4 (and would help server load in the dieing days).
|
| « Last Edit: Jan 20th, 2010, 7:20pm by 99of9 » |
IP Logged |
|
|
|
99of9
Forum Guru
Gnobby's creator (player #314)
Gender:
Posts: 1413
|
|
Re: Will the 2010 Computer Championship be open?
« Reply #127 on: Jan 20th, 2010, 7:59pm » |
Quote Modify
|
Oh dear, I just had a timeout against Opfor2009Fast. The only other bots playing at the same time were: Opfor2008Fast, and Clueless2007Fast (all 3 are single-core as far as I remember).
[EDIT: And another, in fact the humans playing against bots at the moment are having the same problems.]
|
| « Last Edit: Jan 20th, 2010, 8:44pm by 99of9 » |
IP Logged |
|
|
|
Fritzlein
Forum Guru
Arimaa player #706
Gender:
Posts: 5928
|
|
Re: Will the 2010 Computer Championship be open?
« Reply #128 on: Jan 21st, 2010, 4:49am » |
Quote Modify
|
on Jan 20th, 2010, 7:59pm, 99of9 wrote:Oh dear, I just had a timeout against Opfor2009Fast. The only other bots playing at the same time were: Opfor2008Fast, and Clueless2007Fast (all 3 are single-core as far as I remember).
[EDIT: And another, in fact the humans playing against bots at the moment are having the same problems.] |
|
Ah, so I guess it isn't just GnoBot2009 causing server issues. As usual, the problem is less straightforward than my instant analysis. 
|
|
IP Logged |
|
|
|
RonWeasley
Forum Guru
Harry's friend (Arimaa player #441)
Gender:
Posts: 882
|
|
Re: Will the 2010 Computer Championship be open?
« Reply #129 on: Jan 21st, 2010, 6:43am » |
Quote Modify
|
on Jan 20th, 2010, 3:18pm, Fritzlein wrote:
Ron, if you are reading this, what do you think of removing GnoBot2009 from the pool of benchmark bots? |
|
I'll try to follow this. Right now it seems we don't understand what's happening, so I'm not making any rulings.
|
|
IP Logged |
|
|
|
Janzert
Forum Guru
Arimaa player #247
Gender:
Posts: 1016
|
|
Re: Will the 2010 Computer Championship be open?
« Reply #130 on: Jan 21st, 2010, 7:27am » |
Quote Modify
|
Given that the first 3-4 bots are providing essentially no differentiation. I'd rather not reduce the field of competitive opponents even further and would prefer to have gnobot2009blitz stay in at this point.
Janzert
|
|
IP Logged |
|
|
|
aaaa
Forum Guru
Arimaa player #958
Posts: 768
|
|
Re: Will the 2010 Computer Championship be open?
« Reply #131 on: Jan 21st, 2010, 8:54am » |
Quote Modify
|
GnoBot2009Fast, for one, is well known to me as a bot with quite the timeout problem.
|
|
IP Logged |
|
|
|
99of9
Forum Guru
Gnobby's creator (player #314)
Gender:
Posts: 1413
|
|
Re: Will the 2010 Computer Championship be open?
« Reply #132 on: Jan 21st, 2010, 2:02pm » |
Quote Modify
|
on Jan 21st, 2010, 8:54am, aaaa wrote:| GnoBot2009Fast, for one, is well known to me as a bot with quite the timeout problem. |
|
It is certainly susceptible if it doesn't have the whole server to itself. It uses every available resource.
|
|
IP Logged |
|
|
|
jdb
Forum Guru
Arimaa player #214
Gender:
Posts: 682
|
|
Re: Will the 2010 Computer Championship be open?
« Reply #133 on: Jan 21st, 2010, 2:12pm » |
Quote Modify
|
on Jan 21st, 2010, 7:27am, Janzert wrote:Given that the first 3-4 bots are providing essentially no differentiation. I'd rather not reduce the field of competitive opponents even further and would prefer to have gnobot2009blitz stay in at this point.
Janzert |
|
Bot 4 provided some differentiation today. 
|
|
IP Logged |
|
|
|
Fritzlein
Forum Guru
Arimaa player #706
Gender:
Posts: 5928
|
|
Re: Will the 2010 Computer Championship be open?
« Reply #134 on: Jan 21st, 2010, 2:46pm » |
Quote Modify
|
on Jan 21st, 2010, 7:27am, Janzert wrote:| Given that the first 3-4 bots are providing essentially no differentiation. I'd rather not reduce the field of competitive opponents even further and would prefer to have gnobot2009blitz stay in at this point. |
|
That's a good point. Except for one loss by clueless, everyone swept the first four bots. So dropping GnoBot2009 would leave the burden on just three bots, namely Bomb2005Fast, OpFor2009Fast, and Clueless2009Blitz. But those three seem to be doing a good job separating anyone who plays them so far. Marwin nearly swept all three, pragmatic_theory is having trouble beating any of them, and GnoBot is somewhere in between.
The problem with including GnoBot2009Blitz as a benchmark bot is not just that it is disruptive to the server and anyone else trying to play a bot. The problem is also that GnoBot2009Blitz may be adding more noise than signal to the qualifying scores due to performing worse (and possibly even timing out) when it isn't the only bot running.
|
|
IP Logged |
|
|
|
|
Home
Help
Search
Members
Login
Register
Reply
Notify of replies
Send Topic
Print