Author |
Topic: Omar = OmarFast , bot_bomb = bot_spe (Read 7221 times) |
|
99of9
Forum Guru
Gnobby's creator (player #314)
 
Gender: 
Posts: 1413
|
 |
Re: Omar = OmarFast , bot_bomb = bot
« Reply #30 on: Jun 8th, 2005, 9:00pm » |
 Quote  Modify
|
On looking at the results, I think Fritz's scheme is probably a success. One thing to keep in mind as you read this post is how to mix fast and slow ratings for intermediate time controls:
slow fast control
0% 100% 15s or under
20% 80% 30s
40% 60% 1min
60% 40% 2min
80% 20% 4min
100% 0% 8min or over
Here are a few points of analysis which I think are interesting:
1) Some of the most obviously fast/slow oriented people do indeed show up as significantly different on the two scales.
On the "Slow" side we have Omar (+390), Belbo (+42 , Clauchau (+286), and Fotland, bleitner, CeeJay, RonWeasley, ...
On the "Fast" side we have Arimanator (-483), Sip8980 (-210), KamikazeKing (-197), Haizhi (-197), ... (PMertens, Rabbitball, and Aamir Syed are also strongly on this side of the equation)
2) It is interesting that the slow list (as shown in previous post) starts with our two world champions, Fritzleing, Belbo, and follows up with Omar. I agree with this ratings assessment that these 3 players are themost dangerous at long time controls. In fact these players were 3 of the top 4 in the postal tourney. JDB also did very well, but had significantly easier opponents because of his low rating coming in.
3) Don't pay too much attention to the differences in ratings of bots. As Fritz mentioned earlier, because they are usually constrained to only play one time control, only their weighted average at their preferred control matters. But they act correctly at sucking/giving a correct mixture of ratings to people who play poorly/well at their particular time control.
4) Here's the current order of favourites these ratings suggest for WC time controls if we conducted a tournament now (2 min = 60% slow / 40% fast):
Fritzlein
99of9
Belbo
robinson
Omar
bleitner
mouse
5) PLEASE NOBODY TAKE THIS PERSONALLY, YOU MAY NOT LIKE THE EXAMPLE, BUT IT IS THE BEST EXAMPLE AVAILABLE TO ME.
Because this system in some sense keeps track of fast and slow ratings separately, strong "slow" players (eg Omar) already have a low "fast" rating, so are not penalised much when they play blitz games and lose. This means that for example, bomb_blitz doesn't steal all Omar's points, and is instead only moderately rewarded for being able to beat Omar at blitz (which is, at the moment, not such a hard thing to achieve  ). A corrolary is that people who can beat bomb at blitz (eg Arimanator) do not then channel those many points on to themselves. Arimanator earns a reasonable blitz rating for beating a bot which can beat omar at blitz, but does not earn anything on his slow rating for it.
6) An anomaly explained: MrBrain may be expected to have a real rating that is higher on slow than it is on fast. Problem is, he resigned all his postals, so the exact opposite shows up.
7) It's interesting that this effect is bigger than any that Fritz has found in his analysis of the ratings inaccuracies. Perhaps this should be the first thing we focus on in reforming the rating system?
|
| « Last Edit: Jun 8th, 2005, 10:16pm by 99of9 » |
 IP Logged |
|
|
|
jdb
Forum Guru
Arimaa player #214
Gender:
Posts: 682
|
|
Re: Omar = OmarFast , bot_bomb = bot_spe
« Reply #31 on: Jun 8th, 2005, 9:48pm » |
Quote Modify
|
First of all, a fine piece of work. Your analysis was very illuminating for me.
In my opinion, the percentages and cutoffs for the times, look about right. Certainly usable for a trial run.
How does the interpolated rating compare to the rating using only games at a certain time control? For example, Omar's interpolated rating is 1688 for 30 sec games. (Is this correct?) How does this compare to his 30sec rating? Or is it not feasible to compute a rating based only on a certain time control? Could the difference in these ratings be used as a measure of how well the interpolation system works?
|
|
IP Logged |
|
|
|
PMertens
Forum Guru
Arimaa player #692
Gender:
Posts: 437
|
|
Re: Omar = OmarFast , bot_bomb = bot_spe
« Reply #32 on: Jun 8th, 2005, 10:05pm » |
Quote Modify
|
I like it.
Especially the part that people like Omar dont go down in rating for losing 15s games when its absolutely clear how good they really are.
About that sucking from / giving to bots: would it be possible to seperate vbot/vhuman ranking ?
I can bash certain bots in certain proven ways for hours - that does not make me a good player either.
kudos for the work
|
|
IP Logged |
|
|
|
99of9
Forum Guru
Gnobby's creator (player #314)
Gender:
Posts: 1413
|
|
Re: Omar = OmarFast , bot_bomb = bot
« Reply #33 on: Jun 8th, 2005, 10:11pm » |
Quote Modify
|
Yes, that kind of analysis could certainly be used to check how good the interpolation method is (ie Whether performance varies proportional to the logarithm of time. This was Fritz's approximation based on some knowledge of bot performance in chess.)
I guess to be totally accurate though you'd have to base all *opponent's* ratings on only their 30s games as well. Now the problem becomes that people wouldn't have played enough games to establish accurate ratings for the time control. Perhaps there would be sufficient 45s games to try this, I'm not sure.
|
|
IP Logged |
|
|
|
99of9
Forum Guru
Gnobby's creator (player #314)
Gender:
Posts: 1413
|
|
Re: Omar = OmarFast , bot_bomb = bot
« Reply #34 on: Jun 8th, 2005, 11:01pm » |
Quote Modify
|
Paul, Fritz did a fairly complete analysis of that with the current real ratings system, and found that ratings were not strongly affected by vs bot or vs human play. I am of the opinion that once you learn methods to beat bots, you automatically obtain a few tactics you can use against humans, so you are automatically better. In other words, skill is reasonably transferrable. When Fritz arrived on the Arimaa scene, he claimed he would not be good against humans because he had only trained up against David's Bot Offline. Obviously Fritz had learnt a few things... and his rating never stopped rising.
I agree that a player's skill/rating can be a bit different vs bots and vs humans, but then again, my rating vs fritz is different to my rating vs robinson, because I find Robinson's style hard to understand, so he springs more traps on me - even though Fritz is clearly a better player than Robinson against the rest of the pool of players.
Anyway, as requested, here is the same list, this time compiled in a simulation excluding all games vs bots... basically treating them as unrated games. There are certainly differences with the first list, but many of the differences come about from the fact that there are just a whole lot less HvH games played, so the statistics are much more uncertain. Also the humans don't receive the constant inflation we get when newbies arrive, lose points to bots, then quit arimaa (the points are then "redistributed" from the bots by regular players).
SLOW FAST NAME
1845 1946 Fritzlein
1769 1822 99of9
1750 1457 omar
1730 1780 robinson
1701 1594 Spunk
1697 1516 clauchau
1662 1581 Belbo
1647 1401 jdb
1611 1521 rajah
1602 1481 Adanac
1600 1560 Paul
1596 1552 Asturianuco
1594 1534 Groumpf
1589 1559 kerdamdam
1588 1527 qsasha
1585 1527 boris_toplak
1584 1542 siddiqna
1577 1554 dethwing
1570 1527 adsyed01
1566 1519 naveed
1564 1549 Jonathan
1557 1495 Magrathean
1552 1534 YunK
1552 1508 travis
1549 1651 PMertens
1545 1559 spela
1543 1517 xabiron
1543 1517 pikachamp
1543 1517 brad
1537 1523 BLooodyANgel
1536 1524 Yaron
1531 1479 Yuri
1529 1518 asyed94
1529 1509 mouse
1528 1511 fdailey
1526 1504 Blyx
1525 1535 Orc
1522 1561 omarFast
1518 1371 Tore
1512 1500 RonWeasley
1512 1487 BlackKnight
1511 1521 Yusei
1508 1503 Josh
1507 1494 botkiller
1506 1503 ntroncos
1500 1463 ytri
1498 1536 6sense
1497 1687 kamikazeking
1497 1503 WagnerK
1496 1490 jarrausi
1496 1453 antoniotheripper
1495 1468 adannada
1494 1469 Sameer
1494 1516 craig
1494 1497 nbarriga
1494 1470 camperman
1492 1497 cloakski
1492 1446 trevor
1489 1501 OLTI
1488 1422 bleitner
1486 1524 Lucky
1484 1464 Vinvin
1483 1510 dtj
1483 1477 arvindn
1483 1393 Arimanator
1482 1484 gsyed
1481 1473 Moi
1478 1483 Scott
1478 1470 Rileyjal
1477 1466 Ytterbium
1476 1491 Renaissance
1475 1460 BenW
1475 1465 RainMan
1474 1494 Guest207
1471 1466 fotland
1470 1457 TheMadHair
1469 1500 Juha
1469 1359 megamau
1468 1485 AjedrezDude
1466 1453 AmitChaturvedi
1465 1450 kissl
1464 1476 miri
1464 1476 RSA
1464 1476 Robert
1464 1476 Yold
1463 1477 Kobold
1462 1485 jshira
1458 1490 taral
1458 1472 HEx
1457 1483 Kristijan
1457 1483 Keeps
1457 1483 ih8evilstuff
1457 1483 robstar
1457 1483 handsomestofall
1457 1489 xquezme
1456 1484 Guest383
1455 1423 Aaron
1454 1471 v_dhanasekar
1454 1397 Ryan_Cable
1453 1468 Darkenrahl
1452 1488 illz
1452 1481 Arimaardvark
1451 1500 brownsugar726
1450 1435 whiteKnight
1450 1481 ImranG
1449 1499 CeeJay
1448 1492 ugaiW
1448 1480 TipAndMe2
1448 1448 kraj
1444 1458 keita
1442 1500 PatoGuy
1439 1461 carolaina
1434 1458 yarnalito
1433 1470 Nate1729
1430 1350 jemicobel
1429 1447 novicehex
1427 1430 lihanzo
1426 1337 carlsquared
1424 1478 Sophi05
1422 1454 Jimmy_Newtron
1422 1471 blue22
1421 1414 haizhi
1418 1469 naveed4
1416 1468 Greytle
1414 1470 coachbudka
1411 1441 pcpdams
1409 1462 CrazyMan04
1404 1448 Monedero
1397 1429 matjaz
1376 1483 Imran
1361 1465 junaid
1307 1406 gern
1305 1184 Gregorius
1299 1369 eric
1262 1363 Keith
1222 1454 MrBrain
1216 1470 Rabbitball
|
| « Last Edit: Jun 8th, 2005, 11:14pm by 99of9 » |
IP Logged |
|
|
|
PMertens
Forum Guru
Arimaa player #692
Gender:
Posts: 437
|
|
Re: Omar = OmarFast , bot_bomb = bot_spe
« Reply #35 on: Jun 9th, 2005, 1:04am » |
Quote Modify
|
Quote:| Paul, Fritz did a fairly complete analysis of that with the current real ratings system, and found that ratings were not strongly affected by vs bot or vs human play |
|
I know  And the way I remembered it I was on the one extreme side of his chart
Well - since your new calculations make me look even worse than the previous ones ... just forget that I asked for it 
|
| « Last Edit: Jun 9th, 2005, 2:44pm by PMertens » |
IP Logged |
|
|
|
omar
Forum Guru
Arimaa player #2
Gender:
Posts: 1003
|
|
Re: Omar = OmarFast , bot_bomb = bot
« Reply #36 on: Jun 12th, 2005, 11:22am » |
Quote Modify
|
Thanks for trying this out Toby. I was planning to get back to this once we finished the WC format discussion. I would still like to try out the model where the actual time used per move is considered.
If you can send me the code for this I can upload it to the arimaa site so others can download it and try it out also. Or if it's not too long, just post it here.
|
|
IP Logged |
|
|
|
Fritzlein
Forum Guru
Arimaa player #706
Gender:
Posts: 5928
|
|
Re: Omar = OmarFast , bot_bomb = bot
« Reply #37 on: Jun 13th, 2005, 12:25am » |
Quote Modify
|
Thanks a bundle for trying this out! The results don't seem implausible. The big differences in fast and slow ratings for some people give us an incentive to get this sorted out as a high priority for the rating system, probably even higher than the "selection of opponents" problem and the "bots don't learn" problem.
A technical question: Why doesn't bot_Bomb2005Blitz have a slow rating of 1500? If everyone starts with both ratings at 1500, and every game that bot_Bomb2005Blitz plays counts 100% towards fast and 0% towards slow, then shouldn't the slow rating of bot_Bomb2005Blitz remain unchanged throughout?
on Jun 8th, 2005, 9:00pm, 99of9 wrote:
3) Don't pay too much attention to the differences in ratings of bots. As Fritz mentioned earlier, because they are usually constrained to only play one time control, only their weighted average at their preferred control matters. But they act correctly at sucking/giving a correct mixture of ratings to people who play poorly/well at their particular time control.
|
|
Yes, it would seem that _every_ bot should have a "slow" rating lower than its "fast" rating. The fact that this doesn't happen is plausibly attributable to almost every bot playing all its games at the same speed, and that some of the bot games played at varying speeds were actually bot vs. bot games.
But there is one case where there might be enough data for a meaningful test, namely the championship bot Bomb2005. Could you run the program again treating Bomb2005CC and Bomb2005Fast and Bomb2005Blitz all as the same bot playing at three different time controls?
Quote:
7) It's interesting that this effect is bigger than any that Fritz has found in his analysis of the ratings inaccuracies. Perhaps this should be the first thing we focus on in reforming the rating system? |
|
I will repeat the analysis I did when I grouped the games by opponent type (vs. bot / vs. human), only this time I will group the games by speed (30 seconds and under / 45 seconds and over).
|
| « Last Edit: Jun 13th, 2005, 12:26am by Fritzlein » |
IP Logged |
|
|
|
99of9
Forum Guru
Gnobby's creator (player #314)
Gender:
Posts: 1413
|
|
Re: Omar = OmarFast , bot_bomb = bot
« Reply #38 on: Jun 13th, 2005, 1:47am » |
Quote Modify
|
on Jun 13th, 2005, 12:25am, Fritzlein wrote:| A technical question: Why doesn't bot_Bomb2005Blitz have a slow rating of 1500? If everyone starts with both ratings at 1500, and every game that bot_Bomb2005Blitz plays counts 100% towards fast and 0% towards slow, then shouldn't the slow rating of bot_Bomb2005Blitz remain unchanged throughout? |
|
I shared out the extra time due to the initial time reserve and the 60s first move, throughout the game (ie the number of moves in the game matters). Most 15s games actually end up being about 17s games.
Quote:| Yes, it would seem that _every_ bot should have a "slow" rating lower than its "fast" rating. |
|
This phenomena in real ratings would certainly not show up in this system if a bot was rated above 1500 and only ever played 2 min time controls. But it wouldn't adversely affect humans anyway (unless later it suddenly started playing a different time control).
|
| « Last Edit: Jun 13th, 2005, 1:51am by 99of9 » |
IP Logged |
|
|
|
Fritzlein
Forum Guru
Arimaa player #706
Gender:
Posts: 5928
|
|
Re: Omar = OmarFast , bot_bomb = bot
« Reply #39 on: Jun 13th, 2005, 1:55am » |
Quote Modify
|
When I say that every bot should have a slow rating lower than its fast rating, I mean that that's how bots actually perform relative to humans. But I don't expect fast/slow ratings to behave properly when bots only play at a single time control.
I was saying that we shouldn't be alarmed if some bots have a slow rating higher than their fast rating, since playing at only one time control can cause weirdness. Also I was saying that the only good test of whether the system correctly shows that bots are better fast than slow is to run the numbers with all three Bomb2005 bots considered as a single bot. It will be only one data point, but unlike all the other data points, it will be worth something.
|
| « Last Edit: Jun 13th, 2005, 1:56am by Fritzlein » |
IP Logged |
|
|
|
99of9
Forum Guru
Gnobby's creator (player #314)
Gender:
Posts: 1413
|
|
Re: Omar = OmarFast , bot_bomb = bot
« Reply #40 on: Jun 13th, 2005, 2:04am » |
Quote Modify
|
Yes, you are right, that's what I meant by "real fast/slow rating". Sorry I didn't adequately communicate my agreement. I will have a go at bomb sometime - I think your assumption will be shown correct, but there are ways to understand if it isn't.
|
|
IP Logged |
|
|
|
99of9
Forum Guru
Gnobby's creator (player #314)
Gender:
Posts: 1413
|
|
Re: Omar = OmarFast , bot_bomb = bot
« Reply #41 on: Sep 3rd, 2008, 5:10am » |
Quote Modify
|
Just for interest I ran this program over an up to date database, here are the results.
HvH slow ratings
1 chessandgo 2122
2 Fritzlein 2089
3 RonWeasley 1985
4 UltraWeak 1919
5 99of9 1874
6 Adanac 1867
7 arimaa_master 1855
8 jdb 1807
9 blue22 1782
10 mistre 1780
11 clauchau 1778
12 omar 1767
13 PMertens 1735
14 Belbo 1713
15 Spunk 1701
16 robinson 1698
17 Soter 1695
18 mdk 1673
19 Rabbit 1672
20 thorin 1669
21 Swynndla 1659
22 OLTI 1654
23 The_Jeh 1643
24 ArifSyed 1641
25 Brendan 1629
26 Ryan_Cable 1625
27 Arimabuff 1622
28 Tanker_JD 1621
29 woh 1617
30 petitprince 1614
31 rajah 1611
32 IceD 1606
33 xquezme 1603
34 Asturianuco 1596
35 Grey 1596
36 Groumpf 1594
37 JacquesB 1593
38 acroninj 1592
39 Ump 1591
40 qsasha 1588
41 mightybyte 1585
42 boris_toplak 1585
43 siddiqna 1584
44 ChrisB 1583
45 LAbiuso 1583
46 Stonkie 1583
47 xepo 1581
48 DorianGaray 1578
49 dethwing 1577
50 art 1576
51 Tuks 1572
52 adsyed01 1570
53 ArimaaCap 1567
54 kurthyl 1565
55 KT2006 1564
56 Guest5409 1564
57 Jonathan 1564
58 arishiki 1561
59 BlackKnight 1561
60 jonaspojken 1560
61 klabe 1560
62 ttt 1560
63 DanTilkin 1559
64 Magrathean 1557
65 ntroncos 1554
66 YunK 1552
67 KamiKazeKiwi3 1552
68 travis 1552
69 Archonn 1548
70 tize 1548
71 BlackShadow 1546
72 Paul 1545
73 Asubfive 1545
74 rusty 1545
75 Yzaxtol 1545
76 spela 1545
77 brad 1543
78 xabiron 1543
79 pikachamp 1543
80 GordonBlack 1541
81 Tau 1540
82 knarl 1539
83 Gesuma 1538
84 i_am_you 1537
85 gert7 1537
86 glitch 1537
87 ZeroOne 1537
88 marcgb 1537
89 BLooodyANgel 1537
90 Agt 1537
91 Ice 1537
92 Virgeist 1536
93 Raymond 1536
94 Yaron 1536
95 mikbuster 1535
96 Heidissimo 1535
97 kerdamdam 1532
98 nauboone 1532
99 Yuri 1531
100 DieInnereMelone 1530
HvH fast ratings
1 Fritzlein 2075
2 chessandgo 2022
3 PMertens 1854
4 robinson 1838
5 Adanac 1811
6 99of9 1786
7 mdk 1785
8 Soter 1771
9 arimaa_master 1746
10 IceD 1698
11 Brendan 1680
12 Arimabuff 1649
13 Ryan_Cable 1645
14 MrObvious 1641
15 ArifSyed 1640
16 naveed 1639
17 Swynndla 1630
18 kamikazeking 1620
19 Tuks 1612
20 ttt 1608
21 UltraWeak 1606
22 Spunk 1594
23 Guest5409 1585
24 willwould 1585
25 DorianGaray 1585
26 seanmcl 1585
27 Belbo 1583
28 challenger 1582
29 mistre 1579
30 BlackKnight 1570
31 Gesuma 1570
32 xepo 1569
33 Adlai 1568
34 Raymond 1566
35 BlackShadow 1565
36 Yzaxtol 1563
37 omarFast 1561
38 nbarriga 1560
39 spela 1559
40 Gorgapor 1555
41 dethwing 1554
42 jonaspojken 1554
43 Asturianuco 1552
44 Jonathan 1549
45 chessdiva27 1549
46 gert7 1548
47 petitprince 1546
48 Lautresault 1545
49 nauboone 1545
50 tize 1544
51 kurthyl 1543
52 OLTI 1543
53 Kraizy_Dave 1543
54 Sir_Twit 1543
55 sakano 1542
56 siddiqna 1542
57 Tau 1541
58 aXiom 1539
59 DieInnereMelone 1538
60 ntroncos 1537
61 JCricket 1537
62 art 1535
63 emeryaj 1535
64 Rancca 1535
65 Orc 1535
66 megamau 1535
67 Groumpf 1534
68 YunK 1534
69 snetnis 1534
70 xquezme 1532
71 Grey 1531
72 jl_ 1531
73 archigavr 1531
74 BilalQ 1529
75 Garyth 1528
76 ArimaaCap 1527
77 GordonBlack 1527
78 qsasha 1527
79 boris_toplak 1527
80 adsyed01 1527
81 Fleisch 1526
82 Ice 1525
83 knarl 1525
84 Virgeist 1524
85 Lucky 1524
86 Yaron 1524
87 BLooodyANgel 1523
88 LifeBlade 1523
89 i_am_you 1523
90 glitch 1523
91 marcgb 1523
92 ZeroOne 1523
93 smonroy 1522
94 acroninj 1521
95 drleper 1521
96 rajah 1521
97 Yusei 1521
98 mightybyte 1521
99 pikachamp 1517
100 xabiron 1517
|
| « Last Edit: Sep 3rd, 2008, 5:10am by 99of9 » |
IP Logged |
|
|
|
|
Home
Help
Search
Members
Login
Register
Reply
Notify of replies
Send Topic
Print