Author |
Topic: 2009 Arimaa Challenge (Read 7813 times) |
|
99of9
Forum Guru
Gnobby's creator (player #314)
 
Gender: 
Posts: 1413
|
 |
Re: 2009 Arimaa Challenge
« Reply #75 on: Mar 23rd, 2009, 3:24pm » |
 Quote  Modify
|
on Mar 23rd, 2009, 2:02pm, Janzert wrote:| To me the most likely explanation at this point for all of this is a cron job running at 4am on Sunday mornings that is sending a SIGHUP to the bot script. |
|
One minor positive... none of the CC timeouts were caused by this particular issue.
|
|
 IP Logged |
|
|
|
RonWeasley
Forum Guru
Harry's friend (Arimaa player #441)
Gender:
Posts: 882
|
|
Re: 2009 Arimaa Challenge
« Reply #76 on: Mar 24th, 2009, 8:32am » |
Quote Modify
|
on Mar 23rd, 2009, 7:31am, Fritzlein wrote:
Ron, have you ruled on these games yet? Must they also be played out from the point of the timeout? |
|
From the information in this thread, I can't tell yet why these games timed out. Did I miss it? If we can pin it on the network/server, or if we give up without finding the cause, they will be played from the point of timeout.
I would like to appoint Janzert as the expert witness on this technical issue, due to his expertise and that his bot is not one of this year's contenders. If he accepts, I would like him to decide if the timeout cause can be found or we should abandon that search. Other opinions are welcome.
There was mention that there are TWO games where clueless timed out against aaaa? Same colors? Both caused by network/server error at the point of timeout? I had thought there were only one. If there are two, the second game is not authorized as a screening game. The earlier one takes precedence. That game must be continued at the point of timeout. If that game does not get finished then that and the similarly sided game vs GnoBot cannot count. Sorry, aaaa. You are somehow a lightning rod for these timeouts.
|
|
IP Logged |
|
|
|
omar
Forum Guru
Arimaa player #2
Gender:
Posts: 1003
|
|
Re: 2009 Arimaa Challenge
« Reply #77 on: Mar 24th, 2009, 1:14pm » |
Quote Modify
|
Janzert is right. I looked around on the server and there is a root level cron job which rotates the web server log and restarts the web server at 04:02; the same time when these games timed out.
Code:
99969 Sun Mar 15 04:02:03 2009 bot_clueless vs aaaa
100645 Sun Mar 22 04:02:19 2009 ChrisB vs bot_GnoBot
100646 Sun Mar 22 04:02:04 2009 woh vs bot_clueless
|
|
I should have temporarily disabled the cron job and ran the rotation script manually. It has been so long since I set this up, I forgot about it. All these games need to be replayed. However it might get to to be a timing problem for these players to inform me when they are available to continue the game and for me to also be available at that time to restore the game especially when we have less than a week left for the screening period. Also it might become a fairness issue if I am able to coordinate a time with one player, but not another. It might be easier to allow the players to just replay these games from start. I would think that game 100358 aaaa vs bot_clueless should also be treated this way. As coordinator I would like to request the TD to take this circumstance into consideration.
As if this wasn't enough, there was also the problem that the day before yesterday (Sunday) I forgot to enable the CC bots after disabling them to run the post-analysis script for GnoBot. So we lost about 48 hours of time in the two weeks screening period. I would like to request that the screening period be extended by 2 days.
Toby even pinged me and mentioned that I probably forgot to enable the CC bots after the final WC game, but I told him that only the bots on the arimaa.com server were disabled during the final WC game. When he pinged me there was another problem on the server that was preventing players from being able join the game after starting a bot. This was caused by a web server configuration change I made while experimenting with mod_perlite on the server.
|
| « Last Edit: Mar 24th, 2009, 1:38pm by omar » |
IP Logged |
|
|
|
Janzert
Forum Guru
Arimaa player #247
Gender:
Posts: 1016
|
|
Re: 2009 Arimaa Challenge
« Reply #78 on: Mar 24th, 2009, 1:19pm » |
Quote Modify
|
[EDIT: Oops, you can ignore this. Looks like Omar already found the program causing it]
Sure I don't mind, although I think I probably have reached the limit of what can be found from the publicly accessible information.
To summarize; there are current 4 bot timeouts that are still being counted for now (games 99969, 100358, 100645 and 100646). It is pretty clear from the net log that game 100358 timed out because of a network problem.
The other three games are a less conclusive. Looking at the logs for games 99969 and 100645 it at first appears that the bot in each case has crashed mid-search and hence timed out. But then it is seen in game 100646 that the bot interface exits while waiting for the opponents move from the game server. The only reason I can see from looking at the bot interface code that it will do this is if it receives a SIGHUP or SIGINT signal. Being that all three of these timeouts occurred at about the same time on a Sunday morning (4am EST) it seems most likely that they all have the same cause. Also none of the other games that completed successfully seem to have been played at that time regardless of day.
At this point I think the only way to confirm this is if the program Omar uses to start the bot interface, logs the stdout from the interface to a file, or if he can find a cron job or other process that is sending a signal at 4am to the bot interface to close.
Janzert
|
| « Last Edit: Mar 24th, 2009, 1:20pm by Janzert » |
IP Logged |
|
|
|
woh
Forum Guru
Arimaa player #2128
Gender:
Posts: 254
|
|
Re: 2009 Arimaa Challenge
« Reply #79 on: Mar 24th, 2009, 1:33pm » |
Quote Modify
|
on Mar 24th, 2009, 1:14pm, omar wrote:| However it might get to to be a timing problem for these players to inform me when they are available to continue the game and for me to also be available at that time to restore the game especially when we have less than a week left for the screening period. It might be easier to allow the players to just replay these games from start. |
|
Omar, I could replay my game against clueless Friday between 3 and 4PM GMT, Saturday between 7 and 8AM or between 12 and 15PM GMT or Sunday between 7 and 8 AM GMT.
Clueless timed out against me on move 9. So I don't think it makes much difference just to replay this game from the start.
|
|
IP Logged |
|
|
|
99of9
Forum Guru
Gnobby's creator (player #314)
Gender:
Posts: 1413
|
|
Re: 2009 Arimaa Challenge
« Reply #80 on: Mar 24th, 2009, 1:52pm » |
Quote Modify
|
I don't think the games should be switched from "continue" to "replay" just because it's easier for the coordinator to organize. To be brutally honest, since the coordinator was responsible for the faults, he has a responsibility to go to the extra trouble to rectify them as per the TD's ruling. I don't mind if the screening period is extended because of this and the other delays.
|
|
IP Logged |
|
|
|
Janzert
Forum Guru
Arimaa player #247
Gender:
Posts: 1016
|
|
Re: 2009 Arimaa Challenge
« Reply #81 on: Mar 24th, 2009, 2:05pm » |
Quote Modify
|
I don't have strong feelings either way about whether the games should be continued from the current position or replayed from the start.
I do think that continuing BvB games is great as neither bot can gain anything by the interruption anyway (ignoring possible opening book changes or future learning bots). But once a human is involved I think it is impossible for a human to not benefit from the interruption. Even if the human should consciously try to avoid any further analysis of the final position there is a certain amount of subconscious review that will occur regardless.
Janzert
|
|
IP Logged |
|
|
|
99of9
Forum Guru
Gnobby's creator (player #314)
Gender:
Posts: 1413
|
|
Re: 2009 Arimaa Challenge
« Reply #82 on: Mar 24th, 2009, 3:13pm » |
Quote Modify
|
on Mar 24th, 2009, 8:32am, RonWeasley wrote:There was mention that there are TWO games where clueless timed out against aaaa? Same colors? Both caused by network/server error at the point of timeout? I had thought there were only one. If there are two, the second game is not authorized as a screening game. The earlier one takes precedence. That game must be continued at the point of timeout. If that game does not get finished then that and the similarly sided game vs GnoBot cannot count. Sorry, aaaa. You are somehow a lightning rod for these timeouts.
|
|
The first was quite a while ago. At the time we could not detect any network or server error. So although jdb did not think it was clueless' fault, he had no way to object to the result standing. The tournament continued, and aaaa played another game(s) with the other colour. That one also timed out, and was more obviously suspicious. The second game is the one you ruled should be continued (and aaaa has decided not to). Since then Janzert and omar have discovered the 4am thing, which explains the first timeout. So now there are two unfinished games (of different colours). Because they are of different colours, there is no need for you to void one to give the other precedence.
|
|
IP Logged |
|
|
|
Fritzlein
Forum Guru
Arimaa player #706
Gender:
Posts: 5928
|
|
Re: 2009 Arimaa Challenge
« Reply #83 on: Mar 24th, 2009, 3:21pm » |
Quote Modify
|
Extending the screening period for a few days is the easiest decision of the crop of decisions. One could argue that if we didn't extend the time, then we would have fallen short of the commitment to have the bots available for two weeks.
It is trickier to decide which games to invalidate, because I understand that multiple independent issues have thrown games into question, but given the sheer number of issues, I think the presumption at the moment should be against the server and in favor of the bot. That is to say, it seems odd at this point to accept any game in which the bot timed out as a valid result. Which timed out games should be accepted and why? Are there timeouts for which we can prove it was the bot's fault, or merely some timeouts where we can't prove it was the server's fault?
The technical difficulty of resuming a timed out game is a significant argument in my mind. The humans who play the bots are doing the community a favor. It is a much greater hassle to resume a game than to replay it from the start; I would hesitate to put that burden on the volunteers. Incidentally I feel differently about the Challenge itself: I would prefer Challenge games to be resumed in similar circumstances, and would expect the Challenge defenders to go the extra mile to make the continuation happen, and would not feel bad forfeiting a defender who refused to resume.
Within the screening, however, the proposal to forfeit a player who doesn't continue a timed-out game seems to give a large bonus to a bot that was hit by a server problem, because the bot might get a victory simply on the grounds that the human player might be too busy to resume the game. It makes more sense to me to nullify invalid games that aren't later rectified, as if those games had never happened.
Unfortunately, supposing that we nullify non-resumed games, I am very uncomfortable with the implied choice given to the human player. Perhaps I would choose to resume a game I was winning and refuse to resume a game I was losing. That would mean the server timeout seriously advantaged me as a human player. I don't like the situation where the choice of the human player has such a large impact. Therefore, as before, I believe that choice should be taken out of it, and all games which were interrupted by server error should be completely invalidated, and the only option the human player should have is the option to replay from scratch or not replay at all (i.e. the same option to participate or not participate that the human player started with).
I am suggesting a different rule for the screening than for the Challenge per se, but note what I am being consistent about: In neither situation should the human player have a choice. In one case the resumption should be mandatory, and in the other case the invalidation of the game should be mandatory.
Unforeseen circumstances like those we now face force us to choose among the lesser of evils. How to choose? I submit that a couple of important principles are
1) Although both are harmful, throwing out a tainted result does less harm than including it. Admittedly, throwing out any result superficially harms the player who lost or was losing, and superficially helps the player who won or was winning. However, if we assume that the forces that taint results will strike at random, then throwing out tainted results is equally likely to hurt or help each side.
2) The way we resolve server errors should, as much as possible, not give "two chances" based on a decision of the players or of the tournament director. Thus a given reason for invalidating a game must always result in an invalidation, and a given reason for resuming a game must always result in resumption.
|
| « Last Edit: Mar 24th, 2009, 3:30pm by Fritzlein » |
IP Logged |
|
|
|
99of9
Forum Guru
Gnobby's creator (player #314)
Gender:
Posts: 1413
|
|
Re: 2009 Arimaa Challenge
« Reply #84 on: Mar 24th, 2009, 3:47pm » |
Quote Modify
|
on Mar 24th, 2009, 3:21pm, Fritzlein wrote:| It is trickier to decide which games to invalidate, because I understand that multiple independent issues have thrown games into question, but given the sheer number of issues, I think the presumption at the moment should be against the server and in favor of the bot. That is to say, it seems odd at this point to accept any game in which the bot timed out as a valid result. Which timed out games should be accepted and why? Are there timeouts for which we can prove it was the bot's fault, or merely some timeouts where we can't prove it was the server's fault? |
|
POI: there are no longer any timeouts blamed on the bots. Thanks mostly to Janzert, all of them have been clearly identified as either a network or server issue.
I'll comment on your other substantive arguments when I get a chance.
|
| « Last Edit: Mar 24th, 2009, 3:50pm by 99of9 » |
IP Logged |
|
|
|
99of9
Forum Guru
Gnobby's creator (player #314)
Gender:
Posts: 1413
|
|
Re: 2009 Arimaa Challenge
« Reply #85 on: Mar 24th, 2009, 7:16pm » |
Quote Modify
|
on Mar 24th, 2009, 3:21pm, Fritzlein wrote:| The humans who play the bots are doing the community a favor. It is a much greater hassle to resume a game than to replay it from the start; |
|
That is not correct in all cases. For example, ChrisB played 39 moves before the bot timed out. I'd guess that the hassle of finding a suitable time is worth less than over 2.5 hours of wasted play.
Quote:| Incidentally I feel differently about the Challenge itself: I would prefer Challenge games to be resumed in similar circumstances, and would expect the Challenge defenders to go the extra mile to make the continuation happen, and would not feel bad forfeiting a defender who refused to resume. |
|
I agree with you in regard to the Challenge. Perhaps the bot should be given some bonus reserve time as compensation for the human pondering. (Although human pondering should certainly be discouraged.)
Quote:| Within the screening, however, the proposal to forfeit a player who doesn't continue a timed-out game seems to give a large bonus to a bot that was hit by a server problem, because the bot might get a victory simply on the grounds that the human player might be too busy to resume the game. It makes more sense to me to nullify invalid games that aren't later rectified, as if those games had never happened. |
|
I would strongly disagree if these were counted as forfeits.
Quote:| Unfortunately, supposing that we nullify non-resumed games, I am very uncomfortable with the implied choice given to the human player. Perhaps I would choose to resume a game I was winning and refuse to resume a game I was losing. |
|
This is most important for aaaa as he is the only one with two timed out games. (Although because of his concept of fairness, he may actually choose to reverse the favouritism and resume from the weaker position.) This is why I suggested that he be given the resumed game at random.
For others I'm not sure that it matters much more than players who play against one bot then on the basis of the result choose not to play the other bot. That introduces a similar kind of bias.
I'm sympathetic to your argument here, but this was not the decision of the TD earlier in the tournament, so this would be an explicit reversal if applied in 2009, and I'm not sure that is healthy.
Quote:| if we assume that the forces that taint results will strike at random, then throwing out tainted results is equally likely to hurt or help each side. |
|
On average it will not affect one bot relative to the other, and that is what's important in the screening process. But any interruption will help the humans relative to the bots. Replays give the human a chance to learn from their mistakes, continuations give the human a chance to ponder.
Quote:| The way we resolve server errors should, as much as possible, not give "two chances" based on a decision of the players or of the tournament director. Thus a given reason for invalidating a game must always result in an invalidation, and a given reason for resuming a game must always result in resumption. |
|
Yes. And this implies that players should try as hard as possible to fulfill what the TD thinks is fairest, rather than make their own judgement of fairness. Although aaaa has had a very annoying schedule, his reason for not continuing is not related to being annoyed, he is basing it on his own opinion of fairness, which I think is an error [especially since his opinion of what is fair has turned 180 degrees since the chat on the 14th of March]. (But by the way, in this instance, failing to continue his games is actually worse for clueless than not continuing them. Since he has beaten gnobot already, it can only help to give clueless a chance, even if it starts from an inferior position.)
|
| « Last Edit: Mar 24th, 2009, 7:53pm by 99of9 » |
IP Logged |
|
|
|
RonWeasley
Forum Guru
Harry's friend (Arimaa player #441)
Gender:
Posts: 882
|
|
Re: 2009 Arimaa Challenge
« Reply #86 on: Mar 25th, 2009, 7:44am » |
Quote Modify
|
Thank you all for commenting. I think there is enough information here to make rulings.
First, there have been enough technical problems on the network/server side that I am ruling that ALL timeouts in screening games, up to this point, are due to the network/server.
Second, considering the relative advantages of replaying or continuing the timed-out games, I rule that the timed-out games be continued from the position of timeout with clocks set as at the beginning of the bot's move. I realize I have chosen the logistically difficult alternative, but I believe this to be the best representation of a fair game. Bot play was not tainted before the timeout. The human can ponder only at a single position and this advantage does not outweigh the importance to me that there are no takebacks or do-overs. If a human player does not complete such a game, the circumstances being immaterial to this ruling, the game does not count and the same colored game against the other bot does not count. Note that a not-completed game is neither a win or a loss. It is a non-game from the point of view of the screening process. I realize a human may, in theory, decide to not continue, and have the game nullified, in an attempt to manipulate the screening process, but I don't believe that to be a risk this year.
Because of the logistical difficulty of the continuation ruling, the screening period is extend seven (7) days to April 5.
[Edit: Corrected date of extension to April 5]
|
| « Last Edit: Mar 25th, 2009, 7:46am by RonWeasley » |
IP Logged |
|
|
|
Fritzlein
Forum Guru
Arimaa player #706
Gender:
Posts: 5928
|
|
Re: 2009 Arimaa Challenge
« Reply #87 on: Mar 25th, 2009, 1:49pm » |
Quote Modify
|
on Mar 24th, 2009, 7:16pm, 99of9 wrote:| And this implies that players should try as hard as possible to fulfill what the TD thinks is fairest, rather than make their own judgement of fairness. |
|
99of9, making my own judgment of fairness does not preclude me from respecting the decision of the Tournament Director. This is not the first case in which I have disagreed with Ron's decision, and (should he be kind enough to continue to serve) it won't be the last. Just because he is the Tournament Director rather than me does not mean than I can or should turn off my evaluative faculty. Nevertheless, I do respect Ron's decisions, and have always abided by them, and will continue to do so in future events (hopefully many) where he is the Tournament Director. I consider his ruling binding in this case for determining the outcome of the screening games. I encourage all players to continue their timed out games if a time can be arranged with Omar.
on Mar 21st, 2009, 11:43am, aaaa wrote:| Apologies in advance for what can be perceived as a power play, but, although I know I will be stepping on the toes of the tournament director with this, I cannot in good conscience resume this game if the understanding is that this is supposed to be a fair assessment of Clueless's playing strength by matching it against mine. |
|
While I agree with you, aaaa, that Ron's decision was not the fairest possible, I understand that reasonable people may differ about what is fair. No solution is entirely fair, not even replaying every tainted game from the start as you and I would have preferred. I encourage you to continue your participation in the screening in accordance with Ron's ruling. Your games during the screening have been interesting and valuable. It would be a further contribution if you would continue the ones that have been interrupted and that Ron has requested be continued. (now including, apparently, your first game against clueless, in which you were behind when clueless timed out) Please consider that perhaps, in this particular case, the perfect might be the enemy of the good. Resuming your games will, I believe, be much better for the Arimaa community than not doing so.
|
| « Last Edit: Mar 25th, 2009, 1:51pm by Fritzlein » |
IP Logged |
|
|
|
99of9
Forum Guru
Gnobby's creator (player #314)
Gender:
Posts: 1413
|
|
Re: 2009 Arimaa Challenge
« Reply #88 on: Mar 25th, 2009, 2:47pm » |
Quote Modify
|
on Mar 25th, 2009, 1:49pm, Fritzlein wrote:
99of9, making my own judgment of fairness does not preclude me from respecting the decision of the Tournament Director. This is not the first case in which I have disagreed with Ron's decision, and (should he be kind enough to continue to serve) it won't be the last. Just because he is the Tournament Director rather than me does not mean than I can or should turn off my evaluative faculty. Nevertheless, I do respect Ron's decisions, and have always abided by them, and will continue to do so in future events (hopefully many) where he is the Tournament Director. I consider his ruling binding in this case for determining the outcome of the screening games. I encourage all players to continue their timed out games if a time can be arranged with Omar. |
|
I was specifically referring to players. My concern is when players change their participation or actions based on their own concept of fairness, when it is not in accordance with the TD's concept of fairness. Since the TD was chosen as the person the community trusted to make a tourney fair, this type of mutiny is not helpful overall.
Discussing or disagreeing with a decision is perfectly fine, and will probably help shape the rules for future years.
|
| « Last Edit: Mar 25th, 2009, 2:48pm by 99of9 » |
IP Logged |
|
|
|
omar
Forum Guru
Arimaa player #2
Gender:
Posts: 1003
|
|
Re: 2009 Arimaa Challenge
« Reply #89 on: Mar 25th, 2009, 4:43pm » |
Quote Modify
|
Players please send me a message through the Contact page with your preferred times for continuing the game.
Woh, I saw you posting with the preferred times. I will email you to set a time.
The silver server is scheduled to be terminated at the end of this month. So after that time there will only be one server to play on. Also the official challenge match games are scheduled to begin on April 5th, so I will have to stop the screening on April 4th.
Ron, from your ruling I am assuming that game 100123 (bot_clueless vs ChrisB) should also be continued. Please confirm.
|
| « Last Edit: Mar 25th, 2009, 7:30pm by omar » |
IP Logged |
|
|
|
|
Home
Help
Search
Members
Login
Register
Reply
Notify of replies
Send Topic
Print