Author |
Topic: 2009 Arimaa Events (Read 4172 times) |
|
Fritzlein
Forum Guru
Arimaa player #706
Gender: 
Posts: 5928
|
 |
Re: 2009 Arimaa Events
« Reply #30 on: Dec 3rd, 2008, 10:38am » |
 Quote  Modify
|
Interesting, aaaa. Your first two simulations show a much larger difference between triple and quadruple elimination than the last one shows. Is this a fluke or does increasing (spread + uncertainty) actually blur the distinction between the two formats?
|
| « Last Edit: Dec 3rd, 2008, 10:39am by Fritzlein » |
 IP Logged |
|
|
|
Fritzlein
Forum Guru
Arimaa player #706
Gender:
Posts: 5928
|
|
Re: 2009 Arimaa Events
« Reply #31 on: Dec 3rd, 2008, 10:41am » |
Quote Modify
|
on Dec 3rd, 2008, 10:34am, aaaa wrote:| From a cursory glance at the code it appears that it's simply the average distance from every true rating to the best one and that the results don't come in. |
|
Hmmm, but then why would that be consistently lower for quadruple elimination than for triple?
|
| « Last Edit: Dec 3rd, 2008, 10:41am by Fritzlein » |
IP Logged |
|
|
|
aaaa
Forum Guru
Arimaa player #958
Posts: 768
|
|
Re: 2009 Arimaa Events
« Reply #32 on: Dec 3rd, 2008, 10:59am » |
Quote Modify
|
on Dec 3rd, 2008, 10:38am, Fritzlein wrote:| Interesting, aaaa. Your first two simulations show a much larger difference between triple and quadruple elimination than the last one shows. Is this a fluke or does increasing (spread + uncertainty) actually blur the distinction between the two formats? |
|
The hypothesis that immediately comes to mind is that with a larger spread of true ratings, the chance increases significantly that there is one player in the tournament who is much stronger than the rest and will often just run away with the title, regardless of what format may happen to be in use.
on Dec 3rd, 2008, 10:41am, Fritzlein wrote:| Hmmm, but then why would that be consistently lower for quadruple elimination than for triple? |
|
Heh heh, I told you it was from a cursory glance. With some more scrutiny, I now think the average difference is instead with respect to the true rating of the eventual winner.
[EDIT]
<Sigh> I should really take the time to figure things out before spouting off. It now appears to be indeed the (obvious) average difference between the winner and the best player and that includes the case where they happen to coincide.
[/EDIT]
|
| « Last Edit: Dec 3rd, 2008, 11:19am by aaaa » |
IP Logged |
|
|
|
Fritzlein
Forum Guru
Arimaa player #706
Gender:
Posts: 5928
|
|
Re: 2009 Arimaa Events
« Reply #33 on: Dec 3rd, 2008, 4:51pm » |
Quote Modify
|
on Dec 3rd, 2008, 10:59am, aaaa wrote:| The hypothesis that immediately comes to mind is that with a larger spread of true ratings, the chance increases significantly that there is one player in the tournament who is much stronger than the rest and will often just run away with the title, regardless of what format may happen to be in use. |
|
There are three data points that don't form a line, so I still suspect some statistical fluke, but it is very plausible that the more the true ratings are spread out, the less difference it makes what system we use. I guess the scenario I am most interested in is where the participants are close to each other in true strength, but the seeding is essentially random. If each bot can beat each other at least one third of the time, that would be a true strength range of 120 points, plus (say) a range of 240 points inaccuracy added to that to randomize the seeding. That scenario probably is where the difference between FTE and FQE is great, whereas if the true strength had a range of 400 points and there was no inaccuracy in ratings (hence perfect seeding) both would perform pretty well.
|
|
IP Logged |
|
|
|
aaaa
Forum Guru
Arimaa player #958
Posts: 768
|
|
Re: 2009 Arimaa Events
« Reply #34 on: Dec 3rd, 2008, 5:33pm » |
Quote Modify
|
on Dec 3rd, 2008, 4:51pm, Fritzlein wrote:| I guess the scenario I am most interested in is where the participants are close to each other in true strength, but the seeding is essentially random. If each bot can beat each other at least one third of the time, that would be a true strength range of 120 points, plus (say) a range of 240 points inaccuracy added to that to randomize the seeding. |
|
I assume you mean that the rating error should be between -120 and +120, but, just to be on the safe side, I've also added the scenario where it's between -240 and +240:
Code:./run3 'formats/floatTripElim' 1024 4 120 120 1000000000
1 37.4%
2 30.5%
3 19.3%
4 12.8%
average number of rounds = 6.84
average rating from best = 24.6
./run3 'formats/floatQuadElim' 1024 4 120 120 1000000000
1 43.0%
2 25.1%
3 19.6%
4 12.3%
average number of rounds = 8.98
average rating from best = 21.6
./run3 'formats/floatTripElim' 1024 4 120 240 1000000000
1 35.4%
2 28.5%
3 20.6%
4 15.5%
average number of rounds = 6.78
average rating from best = 27.2
./run3 'formats/floatQuadElim' 1024 4 120 240 1000000000
1 41.3%
2 23.8%
3 20.0%
4 14.8%
average number of rounds = 8.97
average rating from best = 23.8
|
|
|
| « Last Edit: Dec 3rd, 2008, 5:39pm by aaaa » |
IP Logged |
|
|
|
omar
Forum Guru
Arimaa player #2
Gender:
Posts: 1003
|
|
Re: 2009 Arimaa Events
« Reply #35 on: Dec 3rd, 2008, 5:51pm » |
Quote Modify
|
on Dec 3rd, 2008, 10:22am, aaaa wrote:If we restrict our attention to the top-4 bots (with all due respect to the developers of the others), we see that the current maximum difference in rating is 160. If for each run, the true rating range is set to this number plus the given rating inaccuracy we get the following:
|
|
Thanks for trying this aaaa. The difference between the two formats continues to get bigger as the number of players decreases. The third comparison seems to suggest that quad elimination is also very sensitive to the initial seeding; though as Karl mentioned this seems to be a bit of a fluke. But also I would suggest not changing two variables at the same time because it makes identifying the source difficult.
I think these simulations show that we can make about as much improvement by having better initial seeding as we can by adding a few more rounds.
The only problem I have with adding more rounds is that I try to do one round per day and pre-schedule them so that people know when to show up if they want to watch the games. Also I can't schedule the next round until the current one is finished. With 11 to 12 rounds I would be cutting it close so that if there was a problem and I had to replay a game, it could cause problems with finishing the tournament as scheduled. With about 8 to 9 rounds I have some leeway for mistakes. I could extend the time for the computer tournament, but the way I have it setup now the first two weeks of the month are for the computer tournament and the second two for the challenge match preliminaries. This works out nicely so that I only have to rent an extra dedicated server (silver.arimaa.com) for one month (this is in addition to renting gold.arimaa.com for 3 months). Adding a few more days and crossing into the next month means you have to pay for another month of server rental when you don't fully use it. Ideally these kind of issues should not come into play and they definitely won't once the computer championship gets important enough that hundreds of people show up to watch, but until then I also have to juggle the practical issues 
Maybe we can get better initial seeding by just using a filter that only looks at bot-bot games.
|
| « Last Edit: Dec 3rd, 2008, 6:08pm by omar » |
IP Logged |
|
|
|
jdb
Forum Guru
Arimaa player #214
Gender:
Posts: 682
|
|
Re: 2009 Arimaa Events
« Reply #36 on: Dec 4th, 2008, 8:06am » |
Quote Modify
|
Another tournament format possibility, floatRRQuadElim. It starts with a round robin tournament, and the losses carry forward into the floating quad elimination phase. Statistics on a small run are presented below.
Modify the formats/floatQuadElim file by adding the following lines and uncommenting them.
Code:
pa, ga, ra, ph, rn = getTournState(ARGV[0]);
#print " # #{pa.length} #{rn} \n"
#
#
#if (rn < (pa.length-1) )
# print "* Calling round robin\n"
# print "* Round #{rn+1}\n"
# result = `formats/roundRobin #{ARGV[0]}`
# print result
# exit
#end
|
|
Code:jeff@quad:~/arimaa_bots/sim$ run3 floatQuadElim 200 6 120 240 100000
./run3 'formats/floatQuadElim' 200 6 120 240 100000
1 28.5%
2 28.5%
3 17.0%
4 13.5%
5 8.5%
6 4.0%
average number of rounds = 10.27
average rating from best = 27.8
jeff@quad:~/arimaa_bots/sim$ run3 floatTripElim 200 6 120 240 100000
./run3 'formats/floatTripElim' 200 6 120 240 100000
1 27.5%
2 27.0%
3 14.5%
4 15.0%
5 9.5%
6 6.5%
average number of rounds = 8.29
average rating from best = 25.7
jeff@quad:~/arimaa_bots/sim$ run3 floatRRQuadElim 200 6 120 240 100000
./run3 'formats/floatRRQuadElim' 200 6 120 240 100000
1 29.5%
2 24.0%
3 15.5%
4 13.5%
5 10.0%
6 7.5%
average number of rounds = 10.16
average rating from best = 27.7
|
|
|
|
IP Logged |
|
|
|
omar
Forum Guru
Arimaa player #2
Gender:
Posts: 1003
|
|
Re: 2009 Arimaa Events
« Reply #37 on: Dec 5th, 2008, 4:16pm » |
Quote Modify
|
Jeff are you sure you posted the right code; floatQuadElim doesn't have such lines in it??
Thanks for trying this out. I like it when more people are able to experiment and contribute their results.
|
|
IP Logged |
|
|
|
jdb
Forum Guru
Arimaa player #214
Gender:
Posts: 682
|
|
Re: 2009 Arimaa Events
« Reply #38 on: Dec 5th, 2008, 6:38pm » |
Quote Modify
|
on Dec 5th, 2008, 4:16pm, omar wrote:Jeff are you sure you posted the right code; floatQuadElim doesn't have such lines in it??
Thanks for trying this out. I like it when more people are able to experiment and contribute their results.
|
|
Sorry, I should have been clearer. In the following block of code, the first line is the only line that exists in the original floatQuadElim file. I added all the commented lines, to create the floatRRQuadElim format. It should be line 40. The commented lines need to be uncommented to run the new format. If you want I can email the file to you.
Code:pa, ga, ra, ph, rn = getTournState(ARGV[0]);
#print " # #{pa.length} #{rn} \n"
#
#
#if (rn < (pa.length-1) )
# print "* Calling round robin\n"
# print "* Round #{rn+1}\n"
# result = `formats/roundRobin #{ARGV[0]}`
# print result
# exit
#end |
|
|
|
IP Logged |
|
|
|
omar
Forum Guru
Arimaa player #2
Gender:
Posts: 1003
|
|
Re: 2009 Arimaa Events
« Reply #39 on: Dec 7th, 2008, 11:21pm » |
Quote Modify
|
Thanks I got it now.
|
|
IP Logged |
|
|
|
Fritzlein
Forum Guru
Arimaa player #706
Gender:
Posts: 5928
|
|
Re: 2009 Arimaa Events
« Reply #40 on: Dec 12th, 2008, 11:37am » |
Quote Modify
|
Omar, have to contemplated moving the scheduling window 24 hours later in the week? If so, do you want to implement the change while the Continuous Tournament is ongoing (to test it) or after the Continuous Tournament is over (to minimize disruption)?
|
|
IP Logged |
|
|
|
omar
Forum Guru
Arimaa player #2
Gender:
Posts: 1003
|
|
Re: 2009 Arimaa Events
« Reply #41 on: Dec 14th, 2008, 7:41am » |
Quote Modify
|
I think it might be too disruptive to make that change in the Continuous Tournament while it is in progress. The slots I've selected for say Sunday would suddenly be treated as slots selected for Monday. So everyone would need to make sure they update their times or they could get scheduled for unexpected times.
Changing the programs to shift by 24 hours is not too bad. It gets complicated for shifts that are not a multiple of 24.
I'll make this change for the 2009 WC tournament first and then change it for the Continuous Tournament when it is not in progress.
|
|
IP Logged |
|
|
|
Fritzlein
Forum Guru
Arimaa player #706
Gender:
Posts: 5928
|
|
Re: 2009 Arimaa Events
« Reply #42 on: Dec 14th, 2008, 9:13am » |
Quote Modify
|
OK, thanks. I think it will result in slightly more games being played a favorable times over all. Since you won't be changing the time slot in the middle of the Continuous Tournament, that gives an extra reason not to have the CT run all the way to the week before the World Championship. If people have two weeks instead of one to get in the right times for the Championship, there is less likely to be any confusion on that score.
|
|
IP Logged |
|
|
|
Fritzlein
Forum Guru
Arimaa player #706
Gender:
Posts: 5928
|
|
Re: 2009 Arimaa Events
« Reply #43 on: Dec 14th, 2008, 3:25pm » |
Quote Modify
|
Now 12 players are signed up for the World Championship, so we are guaranteed at least four rounds of preliminaries. Yay!
|
|
IP Logged |
|
|
|
omar
Forum Guru
Arimaa player #2
Gender:
Posts: 1003
|
|
Re: 2009 Arimaa Events
« Reply #44 on: Dec 14th, 2008, 5:23pm » |
Quote Modify
|
Karl reminded me in the chat that the pairing program that we use for floating elimination does not assign the colors properly and that we had to fix it manually last year. I would like to get it fixed before the tournaments start this year.
The color assignment rule I want to use is:
Within each pairing, the player who has played gold a fewer number of times so far against this opponent will play gold for that game. If this is a tie, then the player who has played gold a fewer percentage of times so far in the tournament will play gold for that game, with ties broken randomly.
I was just looking at the code Paul provided for doing the floating elimination pairings.
http://arimaa.com/arimaa/wc/2009/floatDoubleElim
Off hand I don't see how to change the code to add these color assignment rules. Does anyone have an idea on how this could be done.
|
|
IP Logged |
|
|
|
|
Home
Help
Search
Members
Login
Register
Reply
Notify of replies
Send Topic
Print