Author |
Topic: 2015 Championship changes (Read 5781 times) |
|
aaaa
Forum Guru
Arimaa player #958
Posts: 768
|
 |
Re: 2015 Championship changes
« Reply #15 on: Oct 12th, 2014, 8:18am » |
 Quote  Modify
|
Imagine three players being left with one life each and an equal number of wins. Then whoever gets the bye will not only have to win just one game for the championship instead of two, but would additionally be guaranteed second place. That's excessive.
|
|
 IP Logged |
|
|
|
Janzert
Forum Guru
Arimaa player #247
Gender: 
Posts: 1016
|
|
Re: 2015 Championship changes
« Reply #16 on: Oct 12th, 2014, 5:04pm » |
Quote Modify
|
Yes that would be bad, although how bad would depend on the complete tournament history. We've also yet to see that situation arise in the actual tournaments.
So in the past two years when the rule would make any difference it seems to be better and looking back further I didn't see any instance where it would have had an effect at all.
Janzert
|
|
IP Logged |
|
|
|
Fritzlein
Forum Guru
Arimaa player #706
Gender:
Posts: 5928
|
|
Re: 2015 Championship changes
« Reply #17 on: Oct 12th, 2014, 11:36pm » |
Quote Modify
|
on Oct 12th, 2014, 8:18am, aaaa wrote:| Imagine three players being left with one life each and an equal number of wins. Then whoever gets the bye will not only have to win just one game for the championship instead of two, but would additionally be guaranteed second place. That's excessive. |
|
By that time the three players will have each played ten games or more. We have seen huge strength of schedule differences accumulate in a smaller number of games than that. Indeed, it is entirely possible that the SoS of the top performer will be more than 0.5 expected losses tougher than the SoS of either or both opponents. In the latter case, it is not excessive to give the top performer the bye. On the contrary, since giving him the bye gives each of his opponents 0.5 expected losses, it is possible that the bye-counting-as-a-win is less than he deserves. He ought to get an even bigger reward than needing to win one while the others need to win two. Yes, it would be unfair to count a bye as a win if your scenario arose when all three players had essentially identical SoS to that point, but based on recent experience, substantial inequality seems more likely, and indeed, not counting the bye as a win seems at least as likely to be the more unfair scenario.
|
|
IP Logged |
|
|
|
Fritzlein
Forum Guru
Arimaa player #706
Gender:
Posts: 5928
|
|
Re: 2015 Championship changes
« Reply #18 on: Oct 13th, 2014, 12:31am » |
Quote Modify
|
By the way, I realize that I'm being fuzzy by measuring strength of schedule in terms of "expected losses", since expected win percentage depends on the strength of the player. Of course, if Player A has a lower rating than Player B, he has more expected losses against identical opposition, but identical opponents are by definition an equally difficult strength of schedule.
Even so, I think the concept of expected losses is relevant not only to measure which of two players has had the tougher road, but also to quantify how much tougher. To make the comparison fair, you simply have to hypothesize that the two players being compared are of equal strength.
For example, let's take 99of9 and myself after eight rounds in last year's World Championship. Neither of us had had a bye yet. We were both on the point of elimination with two losses so far. There were seven players left, not three, so the situation is less extreme than proposed above, but of a similar nature.
Let's estimate how "excessive" it would have been to give 99of9 a bye-counting-as-a-win while making me play the next round. (That would have given me the chance of being eliminated and automatically placing behind 99of9 even if he lost the following round so that we finished with an equal number of wins over the board.) Let's average our two seed ratings of 2619 and 2189 to get an assumed strength of 2404 for the two of us. Now, assuming everyone else's seed ratings were correct, how tough had our respective schedules been to that point?
My schedule had been
Opponent Rating Expected Losses
-------- ------ ---------------
CENTAURO 1323 0.002
RmznA . 1820 0.034
SilverMitt 1546 0.007
chessandgo 2448 0.563
clyring . 2174 0.210
supersamu 2279 0.327
99of9 . 2404 0.500
Hippo . 2270 0.316
with a total of 1.959 expected losses
99of9's schedule had been
Opponent Rating Expected Losses
-------- ------ ---------------
aurelian . 1577 0.008
Thiagor . 2271 0.317
browni3141 2487 0.617
odin73 . 2086 0.138
Hippo . 2270 0.316
Tuks . 2249 0.291
Fritzlein 2404 0.500
chessandgo 2448 0.563
with a total of 2.751 expected losses.
Thus if we had given 99of9 a bye-counting-as-win in the next round, whereas you had stuck me with half an expected loss, I still wouldn't have matched the toughness of his schedule. He was in fact granted a bye at that point, and he had totally earned the right for it to count as a win. Giving him a bye that didn't count as a win was grossly unfair.
Yes, I know we have only had the current format for a short time, and that next year the pairings could shake out very differently. Nevertheless, I repeat my claim that it seems at least as likely that giving a bye-counting-as-win in a late round will be a justified reward as that it will be unjustified.
|
| « Last Edit: Oct 13th, 2014, 12:39am by Fritzlein » |
IP Logged |
|
|
|
Janzert
Forum Guru
Arimaa player #247
Gender:
Posts: 1016
|
|
Re: 2015 Championship changes
« Reply #19 on: Oct 13th, 2014, 5:34pm » |
Quote Modify
|
Thanks for the detailed analysis Fritz.
Just looking at both the historical record and a number of test tournaments counting byes as wins certainly seems better. I've also been working on looking at a large number of tournaments using a defined error function.
My current error metric is the root mean square of (1/rank) - (1/target rank), to mirror the error in prize payout. For the target ranking I've been looking at both rank by true rating and rank by uTPR. Does this seem like a reasonable measurement?
I have a long run going now and I've have to run at the moment so I'll post detailed results later.
Janzert
|
|
IP Logged |
|
|
|
Fritzlein
Forum Guru
Arimaa player #706
Gender:
Posts: 5928
|
|
Re: 2015 Championship changes
« Reply #20 on: Oct 16th, 2014, 12:21am » |
Quote Modify
|
on Oct 13th, 2014, 5:34pm, Janzert wrote:| My current error metric is the root mean square of (1/rank) - (1/target rank), to mirror the error in prize payout. For the target ranking I've been looking at both rank by true rating and rank by uTPR. Does this seem like a reasonable measurement? |
|
Hmmm, I don't much like the target rank being determined by player rating. In that case we could minimize the "error" by giving the #1 seed a bye counting as a win every round.  I like uTPR rank as a target much better, but using it as a measuring stick definitely raises the question of why we would rank on wins and byes at all instead of directly using uTPR. Clearly ranking by uTPR will have the least error if we measure error relative to uTPR.
I'm afraid your project of measuring whether it is better or worse to count byes as wins is doomed by a lack of clarity on what is "better". So far, based on our limited experience, uTPR seems like a good guide, but you can bet that if we used it for prize payouts, there would eventually arise a case in which uTPR was clearly unfair, for reasons that we can't clearly anticipate at present. 
|
|
IP Logged |
|
|
|
Janzert
Forum Guru
Arimaa player #247
Gender:
Posts: 1016
|
|
Re: 2015 Championship changes
« Reply #21 on: Oct 16th, 2014, 2:59am » |
Quote Modify
|
Sorry for leaving this hanging, seems there is always another test that can be run.
So for those that don't want to read all details, the overall conclusion is despite a lot of variability treating byes as wins for ranking is certainly better using the measures in the previous post. I'm am definitely planning to switch to it. But it does open up another question regarding tie breaks, see the end of this post.
For each chart the players in a tournament were chosen uniformly from a rating range as shown in the chart title, either 25 or 50 elo points per player. The Players were then seeded in the tournament with a gaussian distribution having a standard deviation as shown, 0, 25, or 500. 1000 tournaments were run for each tournament size for the given settings. I'm sure someone here could tell what the resulting confidence intervals are, but I haven't gone and looked up how to calculate it again. My feeling is that they are still fairly broad though.
The blue line in each chart labeled "Has Effect" is the percentage of tournaments that the rule change makes a difference. The red line, labeled "Better True", is the percentage of times when there is a difference that the bye+win rule performance better as measured against the ranking using the true ratings of the players. The yellow line, labeled "Better TPR", is the same when comparing against the final rankings by uTPR.
Rating range of 50 per player:
Rating range of 25 per player:
As you can see there is quite a bit of variability, particularly an even-odd effect when measured against the true rankings. Also it seems as the player ratings get closer and the seeds get worse it becomes closer to even with the win only rule.
The question now is what to do about tie breaks for 2nd and 3rd place. As was pointed out in the chatroom, the current tie break will never trigger for 2nd place. It seems rather odd to possibly play an extra game to determine 3rd place but not 2nd. I think if we were starting from scratch I would argue for no extra tie breaks at all.
Janzert
p.s. I started this post prior to Fritzlein's post above, and only finally posted it because I really have to get to bed. So I'll try and absorb and respond to Fritzlein's post tomorrow.
|
| « Last Edit: Oct 16th, 2014, 8:53am by Janzert » |
IP Logged |
|
|
|
Janzert
Forum Guru
Arimaa player #247
Gender:
Posts: 1016
|
|
Re: 2015 Championship changes
« Reply #22 on: Oct 16th, 2014, 12:21pm » |
Quote Modify
|
on Oct 16th, 2014, 12:21am, Fritzlein wrote:
Hmmm, I don't much like the target rank being determined by player rating. In that case we could minimize the "error" by giving the #1 seed a bye counting as a win every round. |
|
Insomuch as you trust the seed yes. 
Quote:| I'm afraid your project of measuring whether it is better or worse to count byes as wins is doomed by a lack of clarity on what is "better". |
|
Yes this is certainly the crucial element, how to decide what is better. The reason I chose to look at true ranking and tpr ranking is that for me they are two important aspects that, while not directly in competition, often end up being oppositely favored when given other constraints. They also seem to be fairly representative of two different philosophies for what the tournament should be*. Happily in this case the change tends to improve both.
Of course if someone else has a metric that I can implement easily I'd be happy to take a look at it as well. Although I'm about to leave town for a few days so it probably won't be until next week.
While I don't mind continuing a conversation on the topic, I guess at this point unless someone comes up with surprising new information I'm pretty well decided on this change. So I'd really like to hear more on what to do with the tie break system.
Janzert
|
| « Last Edit: Oct 16th, 2014, 12:24pm by Janzert » |
IP Logged |
|
|
|
Fritzlein
Forum Guru
Arimaa player #706
Gender:
Posts: 5928
|
|
Re: 2015 Championship changes
« Reply #23 on: Oct 17th, 2014, 12:07am » |
Quote Modify
|
It is indeed promising that the change improves performance on two very different metrics. As for playoff games, I'm happy to do without. The round of elimination would be the primary ranking, with ties broken by UTPR, including a potential tie for third/fourth place. But I also wouldn't mind retaining a playoff game if third/fourth are eliminated in the same round.
It doesn't seems weird to me to have a playoff to break a tie for third and no playoff for second place. Lots of sports competitions have a built-in third-place game and never have a second-place game, because the standard single-elimination format never leaves second place in doubt. If our triple-elimination format also never leaves second place in doubt, what is the problem?
I don't think the lack of a playoff for second/third should be an argument against the proposed format until such time as the format gets it clearly wrong, i.e. the format gives second place to someone who clearly performed worse than the third-place finisher.
|
|
IP Logged |
|
|
|
Janzert
Forum Guru
Arimaa player #247
Gender:
Posts: 1016
|
|
Re: 2015 Championship changes
« Reply #24 on: Oct 20th, 2014, 11:19am » |
Quote Modify
|
Once I started writing the modified rule out and realizing that defining it as "reverse order of elimination" seemed like the natural way to state it, then the playoff game just for third place feels fine as well.
The modified version is now in the rules. Let me know if I got something wrong.
Janzert
|
|
IP Logged |
|
|
|
Janzert
Forum Guru
Arimaa player #247
Gender:
Posts: 1016
|
|
Re: 2015 Championship changes
« Reply #25 on: Oct 20th, 2014, 2:28pm » |
Quote Modify
|
One more thing I don't believe has had any discussion but has crossed my mind a few times. I think it might be established enough now to switch out WHRH for WHRE in the seeding formula.
Of the finalists last year only one (odin73) does not currently have at least 20 event games, and in the top 30 there are only 3 (adding RmznA and ikalyoncu).
Janzert
|
|
IP Logged |
|
|
|
Janzert
Forum Guru
Arimaa player #247
Gender:
Posts: 1016
|
|
Re: 2015 Championship changes
« Reply #26 on: Oct 24th, 2014, 6:25pm » |
Quote Modify
|
Since it came up in the chatroom and looking back it appears I've never actually posted a confirmation. I will be able to take on the tournament director position again this year. Omar has also given his approval for me to do so.
Janzert
|
|
IP Logged |
|
|
|
Fritzlein
Forum Guru
Arimaa player #706
Gender:
Posts: 5928
|
|
Re: 2015 Championship changes
« Reply #27 on: Oct 25th, 2014, 1:03am » |
Quote Modify
|
on Oct 24th, 2014, 6:25pm, Janzert wrote:Since it came up in the chatroom and looking back it appears I've never actually posted a confirmation. I will be able to take on the tournament director position again this year. Omar has also given his approval for me to do so.
Janzert |
|
This is excellent news for the Arimaa community! (I didn't celebrate earlier because I didn't stop to think what would have happened if you were unable or unwilling to be TD again. <shudders>)
|
|
IP Logged |
|
|
|
Janzert
Forum Guru
Arimaa player #247
Gender:
Posts: 1016
|
|
Re: 2015 Championship changes
« Reply #28 on: Oct 28th, 2014, 1:06pm » |
Quote Modify
|
on Oct 20th, 2014, 2:28pm, Janzert wrote:| I think it might be established enough now to switch out WHRH for WHRE in the seeding formula. |
|
So the feeling I'm getting from here and the chatroom is that no one cares in either direction on this, or possibly I've burned everyone out on rule changes.
I'm pretty much right on the cusp between WHRE has the coverage needed and give it one more year so we can get a data point from the WC to check if it has enough coverage.
Janzert
|
|
IP Logged |
|
|
|
Janzert
Forum Guru
Arimaa player #247
Gender:
Posts: 1016
|
|
Re: 2015 Championship changes
« Reply #29 on: Nov 2nd, 2014, 9:43pm » |
Quote Modify
|
Given that I have no strong feeling either way I decided to leave it the same for now and stuck with WHRH.
Unless something else comes up I think the rules are now finished for this year.
Janzert
|
|
IP Logged |
|
|
|
|
Home
Help
Search
Members
Login
Register
Reply
Notify of replies
Send Topic
Print