Author |
Topic: WCCC Protest (Read 4672 times) |
|
jdb
Forum Guru
Arimaa player #214
Gender: 
Posts: 682
|
 |
WCCC Protest
« on: Jan 23rd, 2006, 6:29pm » |
 Quote  Modify
|
I sent this email to Omar. It was suggested I post a copy of the email in the forum.
Quote:Hi Omar,
As Clueless's manager, I wish to lodge a protest concerning the pairings. I think it is unfair for Clueless to have to play again in round 6. Clueless has played Bomb twice and Aamira once. While Aamira and Bomb have not played at all.
I don't know how to contact the tournament director, so if you could contact him for me, it would be much appreciated.
I can go into more detail regarding why I feel the pairings are unfair, if requested.
Thanks for your time
Jeff |
|
|
|
 IP Logged |
|
|
|
Fritzlein
Forum Guru
Arimaa player #706
Gender:
Posts: 5928
|
|
Re: WCCC Protest
« Reply #1 on: Jan 24th, 2006, 11:04am » |
Quote Modify
|
Now that Clueless has beaten Aamira in round six, the protest gains even more force. Aamira's claim to the seventh-round bye rests on having a higher pretournament ratings (as per the tournament rules), whereas Clueless' claim to the bye rests on Bomb vs. Aamira being a first-time matchup as opposed to Bomb vs. Clueless being a third-time matchup.
The non-repeat matchup should always take precedence in assigning byes between players with an equal number of byes, regardless of W-L record. This time, however, W-L record is also tied at 3-2! We should not let a broken pairing algorithm subvert the intent of the pairing scheme. JDB, I hope Don rules in your favor, at least for round seven.
|
|
IP Logged |
|
|
|
doublep
Forum Guru
Badger author
Gender:
Posts: 82
|
|
Re: WCCC Protest
« Reply #2 on: Jan 24th, 2006, 2:08pm » |
Quote Modify
|
Just to note, I don't mind a rule change even right in the middle of the tournament. Aami-ra was already too lucky to have two games against Loc, and now it has lost both games to Clueless, so it just doesn't deserve the second place. Let it lose to Bomb 
|
|
IP Logged |
|
|
|
Ryan_Cable
Forum Guru
Arimaa player #951
Gender:
Posts: 138
|
|
Re: WCCC Protest
« Reply #3 on: Jan 24th, 2006, 4:12pm » |
Quote Modify
|
Thank you for this very sporting statement doublep. I am very happy to be able to avoid having the tournament marred by a messy dispute. With the concession that Clueless deserves second place unless Aamira beats Bomb, I think it becomes a moot point which bot plays Bomb tonight and which bot plays Bomb in the morning.
Also, I think you should be quite proud of a third place finish from a bot that wasn’t even in the last CC. And a look through their game records shows Clueless was much luckier to get both of its wins than is apparent from the tournament games.
|
|
IP Logged |
|
|
|
omar
Forum Guru
Arimaa player #2
Gender:
Posts: 1003
|
|
Re: WCCC Protest
« Reply #4 on: Jan 24th, 2006, 4:42pm » |
Quote Modify
|
Sorry I did not see Jeff's posting or email until this afternoon.
I had sent Jeff the following reply by email:
Quote:
Hi Jeff,
Im sorry I did not see this message until Tuesday afternoon.
However the pairing is done using the Floating Triple Elimination program we developed last summer. It is completely deterministic and only the color assignment is sometimes done randomly. The tournament director who is David Levy of the ICGA would only be consulted if there is a dispute about the outcome of a game. He is not involved with the pairing or color assignments.
Since we have started the tournament using the FTE
program for doing the pairing and color assignent, we should stick with it for the remainder of the tournament even if it sometimes does not do the optimal assignments. However, we should discuss this in the forum so that we can improve the program for future tournaments.
Omar
|
|
Jeff replied back that since we are using a relatively untested, newly developed, pairing system we should consider intervention since the situation appears to be clearly unfair.
I have decided to pause the next round until we resolve this issue. I will consult David Levy for his decision on whether or not we should intervene in this situation. I would also like to invite the Arimaa community to post their opinion on this issue. I will refer David to this thread when consulting him.
My own opinion now is that I think we should intervene and the round 7 game should be:
Bomb vs Aamira
|
|
IP Logged |
|
|
|
99of9
Forum Guru
Gnobby's creator (player #314)
 
Gender:
Posts: 1413
|
|
Re: WCCC Protest
« Reply #5 on: Jan 24th, 2006, 5:39pm » |
Quote Modify
|
The question is, which bot deserves the next bye?
The tournament records so far:
Clueless:
beat Aamira
bye
beat Gnobot
lost Bomb
lost Bomb
beat Aamira
Aamira:
lost Clueless
beat Loc
beat Loc
beat Gnobot
bye
lost Clueless
I've crossed out the results that are the same for both bots (disregarding which colour they played).
If we didn't know how good Loc and Bomb were, this record would basically come down to "Clueless wins head to head but loses to the other bot it plays, and vice versa for Aamira". Because bot results are often not transitive (ie you often find A beats B, B beats C, and C beats A), then both records would be basically equivalent.
In Aamira's favour:
* higher pretournament rating
* gets bye according to tourney rules
* according to pretournament ratings we would think that loc is a harder opponent than bomb
In Clueless' favour:
* head to head wins are better discriminators and hence more valuable than wins against others
* we think from the tournament results that bomb is a harder opponent than loc
I think it is such an unclear situation that if they both lose their last game against bomb, the 2nd place prizemoney should be split.
|
|
IP Logged |
|
|
|
fotland
Forum Guru
Arimaa player #211
Gender:
Posts: 216
|
|
Re: WCCC Protest
« Reply #6 on: Jan 24th, 2006, 11:20pm » |
Quote Modify
|
I'm happy with whatever Omar decides. I would ignore the pre-tournament ratings, since the bot's ratings move around so much depending on who is playing them, and since Bomb was broken.
|
|
IP Logged |
|
|
|
Ryan_Cable
Forum Guru
Arimaa player #951
Gender:
Posts: 138
|
|
Re: WCCC Protest
« Reply #7 on: Jan 25th, 2006, 12:48am » |
Quote Modify
|
on Jan 24th, 2006, 5:39pm, 99of9 wrote:The question is, which bot deserves the next bye? ...
If we didn't know how good Loc and Bomb were, this record would basically come down to "Clueless wins head to head but loses to the other bot it plays, and vice versa for Aamira". Because bot results are often not transitive (ie you often find A beats B, B beats C, and C beats A), then both records would be basically equivalent.
In Aamira's favour:
* higher pretournament rating
* gets bye according to tourney rules
* according to pretournament ratings we would think that loc is a harder opponent than bomb
In Clueless' favour:
* head to head wins are better discriminators and hence more valuable than wins against others
* we think from the tournament results that bomb is a harder opponent than loc
I think it is such an unclear situation that if they both lose their last game against bomb, the 2nd place prizemoney should be split. |
|
This is one of many reasons why I prefer to go with the letter of the rules when they are unambiguous. Once we bring in our belief that Loc was the weakest bot in the tournament and would have been crushed by Clueless to claim that Clueless has been treated unfairly, the tournament starts to be reduced to a beauty contest.
In the WC, we had upsets and near upsets in games with 300 to 500 point favorites. Neither first, second, or third went to Fritzlein and 99of9 though even the most pessimistic gave them 75%+ to win. In this tournament, we had the rather surprising double upset of Aamira by Clueless. Even though I think Bomb is 98%+ to go undefeated, I wouldn’t want to just give Bomb first place today.
Still, Clueless manifestly outperformed Aamira despite unlucky pairings, and based on doublep’s statement above, it doesn’t seem like there is any dispute left to resolve.
|
|
IP Logged |
|
|
|
RonWeasley
Forum Guru
Harry's friend (Arimaa player #441)
Gender:
Posts: 882
|
|
Re: WCCC Protest
« Reply #8 on: Jan 25th, 2006, 8:41am » |
Quote Modify
|
We should be very reluctant to deviate from the agreed tournament rules since the precedent can have an effect on the reputation of arimaa and future sponsorship of the arimaa challenge. We need to assure the muggle world that our promises will be kept so that the motivation to develop bots remains strong.
However, here we have a case where the interested parties, and most spectators, agree that the current rules, without intervention, are not serving their intended purpose. Current rules do allow an appeal to the tournament director. At this point, abiding by his decision also falls within the current rules and we can proceed. As his decision will set a precedent, which may or may not be followed in the future, we should make note of what was important in making the decision, such as, if the change is made, the fact that all particpants agreed to the change. The point is to protect the integrity of the tournament administration.
|
|
IP Logged |
|
|
|
jdb
Forum Guru
Arimaa player #214
Gender:
Posts: 682
|
|
Re: WCCC Protest
« Reply #9 on: Jan 25th, 2006, 8:51am » |
Quote Modify
|
99of9 said:
Quote:| ... Because bot results are often not transitive (ie you often find A beats B, B beats C, and C beats A), ... |
|
Upon further reflection, I think this is the root of the problem.
One could even argue that the pairings are unfair to Aamira. Consider what happens if Aamira is able to beat Bomb?
|
|
IP Logged |
|
|
|
Fritzlein
Forum Guru
Arimaa player #706
Gender:
Posts: 5928
|
|
Re: WCCC Protest
« Reply #10 on: Jan 25th, 2006, 11:22am » |
Quote Modify
|
on Jan 24th, 2006, 5:39pm, 99of9 wrote:| The question is, which bot deserves the next bye? |
|
I don't think that this is the correct question. Yes, it is a consideration which bot is most deserving, but that is of less importance than which pairing is best.
Quote:
If we didn't know how good Loc and Bomb were, this record would basically come down to "Clueless wins head to head but loses to the other bot it plays, and vice versa for Aamira". Because bot results are often not transitive (ie you often find A beats B, B beats C, and C beats A), then both records would be basically equivalent. |
|
The records aren't quite equivalent, because we know (within the tournament) that Bomb has beaten Loc. But again, it isn't of primary importance which of the two bots has the better record. Even if Aamira had a better won-loss record, the next match should be Bomb-Aamira.
The fact that domination isn't transitive is a strong argument in favor of avoiding repeat matchups whenever possible. If there is some kind of fluke that determines the outcome of games between two particular bots, it shouldn't be exaggerated by playing over and over. Imagine, for example, that Bomb had lost its game with Loc, and that Bomb could be beaten easily by Aamira. Imagine that Clueless was the strongest bot except that it was losing every game to Bomb due to some quirk. That would make an even stronger case against Bomb playing Clueless for a third time before playing against Aamira once.
The idea "first decide who deserves a bye, second look for the best pairing among the others" is an artifact of my dumb algorithm, which is sadly part of the rules. A much more desirable view is "Find the best pairing (including who gets the bye) among all available parings". Yes, I wouldn't let any player get two byes more than another, but beyond that avoiding repeat matchups is of much greater importance than deciding who is deserving of a bye.
Whatever Levy rules (and I do see the merit in sticking with a bad algorithm because it's the rule in force), I will argue strenuously that future floating elimination be done on the basis of "best pairing" rather than on the basis of "most deserving of a bye".
|
|
IP Logged |
|
|
|
acheron
Forum Full Member
Arimaa player #1613
Gender:
Posts: 11
|
|
Re: WCCC Protest
« Reply #11 on: Jan 25th, 2006, 3:33pm » |
Quote Modify
|
Quote:| If there is some kind of fluke that determines the outcome of games between two particular bots, it shouldn't be exaggerated by playing over and over. |
|
This is the key point that I believe should determine the outcome in this matter. Pairings should be selected to provide the largest possible crossover, thereby demonstrating which bots are effective in a general way, rather than a singly-targeted effectiveness that allows them to defeat a single computer opponent. such single-target effectiveness, does not demonstrate any greater skill which will enable it to be effective against human players, and as such I'd rather see its impact minimized in pairings.
|
| « Last Edit: Jan 25th, 2006, 3:33pm by acheron » |
IP Logged |
|
|
|
omar
Forum Guru
Arimaa player #2
Gender:
Posts: 1003
|
|
Re: WCCC Protest
« Reply #12 on: Jan 25th, 2006, 4:43pm » |
Quote Modify
|
Thanks everyone for providing feedback on this issue.
I just received a reply from David Levy stating:
Quote:
I agree that Bomb should play Aamria in round 7.
I feel that the algorithm is wrong - ties shoudl be broken in favour of (i.e. give the next bye to) the program with the most losses so far (not the fewest) and further ties broken in favour of the lowest pre-tournament rating.
|
|
That's not quite the reason I was expecting, but anyways, it settles the issue.
I have manually changed the pairing for round 7 to be bot_Bomb vs bot_Aamira with bot_Clueless getting the bye. I've scheduled the game for tomorrow morning since there would not be much notice time if the game was started this evening.
I'd like to thanks David for giving a quick and definite decision which resolved this issue.
|
|
IP Logged |
|
|
|
Ryan_Cable
Forum Guru
Arimaa player #951
Gender:
Posts: 138
|
|
Re: WCCC Protest
« Reply #13 on: Jan 25th, 2006, 7:44pm » |
Quote Modify
|
Wow, I am just stunned!  I hope this is enough to scare us into writing up ironclad rules, and following them to the letter next year. If we had followed David Levy’s system in the WC, robinson would have had to win seven games if he went undefeated:
Round 5
Adanac - bye
robinson - PMertens
Fritzlein - 99of9
Round 6
Fritzlein/99of9 - bye
robinson - Adanac
Round 7
robnson - Fritzlein/99of9
But worse still, if in Round 5 PMertens beat robinson and Fritzlein beat 99of9, robinson would have to face PMertens again.
Round 6
Fritzlein - Adanac
robinson - PMertens
Round 7
Adanac/Fritzlein - PMertens/robinson
Perhaps the most horrible thing is that Adanac would be able to win the Championship with only 5 wins as a reward for being the lowest rated looser.
Oh well, we are going to change the pairing algorithm before the next WC anyway. I am just glad we can come away from this dispute without any hurt feelings.
|
|
IP Logged |
|
|
|
Fritzlein
Forum Guru
Arimaa player #706
Gender:
Posts: 5928
|
|
Re: WCCC Protest
« Reply #14 on: Jan 26th, 2006, 12:49pm » |
Quote Modify
|
Good points, Ryan. It worked very well in the WC to give the bye to the undefeated player rather than the lowest loser. When I argued for the most deserving player to get the first bye, it was because I was afraid of the most deserving player never getting a bye while others did, which would be a bit silly.
Currently my attitude is that repeated matchups are a great source of unfairness, and need to be eliminated whenever possible, even if that means a more deserving player might need an extra win to get through. However, I'm not sure I would still like it if it meant that one player needed two wins more than another to win it all. I wonder whether my latest proposal is vulnerable to this problem, i.e. whether considering W-L record for the bye only after considering eliminating repeat pairings could also mean that one player has a path to victory with two wins fewer than another player.
I totally agree that it would be good to hash this out more to make sure our rules are ironclad before the next elimination tournament. I guess we'll just have to run lots of sample scenarios, and/or try to create counter-examples in which a given system of rules is unfair.
|
| « Last Edit: Jan 26th, 2006, 12:50pm by Fritzlein » |
IP Logged |
|
|
|
|
Home
Help
Search
Members
Login
Register
Reply
Notify of replies
Send Topic
Print