Author |
Topic: 2010 Bot Ladder (Read 5758 times) |
|
omar
Forum Guru
Arimaa player #2
Gender:
Posts: 1003
|
|
2010 Bot Ladder
« on: Sep 18th, 2010, 9:49am » |
Quote Modify
|
I've been meaning to change the bots on the ladder and reset the ladder so we can all have fun clearing it again. To help me decide what bots to put on the ladder, please post suggestions on what bots you would like to see; how many bots you think should be on the ladder, etc.
|
|
IP Logged |
|
|
|
clojure
Forum Guru
Arimaa player #5004
Gender:
Posts: 207
|
|
Re: 2010 Bot Ladder
« Reply #1 on: Sep 18th, 2010, 10:30am » |
Quote Modify
|
I just had this discussion in the chat room. Maybe it helps? Code: 06:00:15 clojure what i dislike strongly about the bot ladder is that in the middle you hit your head against fast/blitz bots. why not make a ladder of CC only? my progress halted totally since i don't want to play fast 06:01:06 novacat I was not aware anyone actually thought that was true, although I guess it is not surprising 06:01:33 aaaa Actually, people don't like the CC bots because the games take so long. 06:03:55 hanzack what does CC stand for? 06:04:12 novacat computer championship 06:04:28 clojure aaaa, it's pretty normal to have long games in go for example. so maybe it's different for chess players? 06:04:44 hanzack cclemon? 06:05:03 clojure i feel that the additional time has really big impact on the quality of games 06:05:19 clojure i got immersed totally different way than in blitz 06:05:48 novacat I agree, but the bot ladder does not need to reflect only that side 06:05:52 clojure and it's even more unfortunate for to watch live games when they are not slow... it's hard to know anything but very superficially 06:06:03 clojure maybe not, that's true 06:06:07 aaaa Maybe introduce 1-minute versions of the bots. 06:06:45 clojure i don't really mind what time bot takes unless i can have as much as i like 06:07:06 aaaa It should not be too easy to reach the top of the ladder. It should still represent an accomplishment. I lean against dropping the blitz bots. 06:07:10 clojure if bot plays better than me, i can learn, whether bot played at 5 sec or 2 min 06:10:25 novacat I am not good at blitz, but I did learn a lot from playing many blitz games (49 games to beat Bomb2005Blitz) 06:10:27 clojure maybe the problem is that there's a group of fast bots in sequence, so you cannot progress at all, and play them later. a better alternative could have cc/fast/blitz in every third place and thus if you always won the cc version, you got to the top but haven't finished the ladder yet 06:10:51 clojure one problem of a blitz is that of interaction with mouse is cumbersome 06:11:20 clojure the physical act of making move is too big a part of the time 06:11:46 novacat that is why there must be a plan of action 06:12:09 clojure i don't get how do you do that in a complex local situation 06:12:13 novacat knowing what moves you will start with will give you a few more seconds later 06:12:31 aaaa Well, that's how you'll develop an instinct for the tactical side of the game. 06:13:32 clojure ok, i'll have more tries then 06:13:35 novacat you also quickly learn what moves don't work, and you can get in 5 + games in an hour 06:14:12 clojure i'm not at this point so sure how important the quantity of games is for arimaa 06:14:32 clojure i rather think that the time spent alone is important 06:14:43 novacat quantity helps learn the basics, quality helps improve them 06:15:39 clojure i personally find that i don't learn much with quick play 06:16:14 novacat what I was really saying, is that if you make a bad move on move 14, you didn't just waste 30 min, but more like 6 06:16:25 clojure but sure experts in go suggest to play both blitz and slow 06:17:06 clojure if you make a bad move in slow game, it's not blunder probably 06:17:12 clojure so you'll learn something really new 06:17:43 clojure besides making bad move forces you to learn to cope as an underdog 06:18:01 clojure you are forced to make the situation complex for the opponent 06:18:32 clojure as in chess you wouldn't want to trade pieces to simplify, and in go you would make fierce tactical battles 06:20:41 novacat If you only play postal, there is plenty of time to consider situations and blitz will not help you much 06:20:43 clojure this was just some feedback on the current system. it's not a big deal 06:21:44 clojure it's funny that i actually play in postals faster than in CC 06:21:48 novacat if you play at 1 minute, it is good to trim the number of options you consider so you don't have time crunches |
| Is there anyone feeling the same way?
|
|
IP Logged |
|
|
|
Sconibulus
Forum Guru
Arimaa player #4633
Gender:
Posts: 116
|
|
Re: 2010 Bot Ladder
« Reply #2 on: Sep 18th, 2010, 12:55pm » |
Quote Modify
|
I think that adding the CC bots would cause most people to never finish the ladder, not because they can't, but because they don't wish to expend the two hour time investment. Bots playing with time controls use the full time allotment, unlike the p1 and p2 bots which move in only a few seconds, but still give you the full two minutes, to use or ignore as you see fit (I prefer ignore).
|
|
IP Logged |
|
|
|
Eltripas
Forum Guru
Meh-he-kah-naw
Gender:
Posts: 225
|
|
Re: 2010 Bot Ladder
« Reply #3 on: Sep 18th, 2010, 2:20pm » |
Quote Modify
|
No CC bots on the ladder please, those games take an eternity not all the people have the time to play those, I think we should keep the bots that are on the ladder plus PT, Marwin and Badger also updating Clueless, Gnobot, Sharp and Opfor would be nice but the addition of the new bots is more important for me, I like the current time controls for p1 and p2 bots, also I think that the existence of fast and blitz bots on the ladder is ok, is not like I'm asking to put lightning bots on the ladder.
|
|
IP Logged |
|
|
|
Fritzlein
Forum Guru
Arimaa player #706
Gender:
Posts: 5928
|
|
Re: 2010 Bot Ladder
« Reply #4 on: Sep 18th, 2010, 4:07pm » |
Quote Modify
|
on Sep 18th, 2010, 10:30am, clojure wrote:Is there anyone feeling the same way? |
| Yes, there are people who feel the same way that you do, i.e. that playing fast and blitz bots is less fun and not the best way to improve. However, you are in the minority. We have experience to prove it. Originally there were CC bots on the ladder, and many people gave up on the ladder because of it. They only wanted games that they could finish quickly. Now that the CC bots are off the ladder, replaced by fast bots, there is a different class of people who give up on the ladder, but not as many as before. This isn't a theoretical argument about what kind of ladder is better, just a practical observation about what the masses want. The "careful thinking" crowd is smaller than the "fast action" crowd. It is what it is.
|
|
IP Logged |
|
|
|
clojure
Forum Guru
Arimaa player #5004
Gender:
Posts: 207
|
|
Re: 2010 Bot Ladder
« Reply #5 on: Sep 18th, 2010, 5:06pm » |
Quote Modify
|
One must notice that careful adjustation of the bot ladder gives different class of players a longer run in the ladder. This is due to the rule that three (3) of the next bots are playable. So the problem is mainly incidental: there is a sequence of fast/blitz bots in the middle of the ladder. If every 3rd bot was at least a bit slower than fast, some people might get further, and it wouldn't hinder those that don't like to waste their time on bots. But I can fare without the ladder. I'm just saying how I feel about it.
|
|
IP Logged |
|
|
|
Fritzlein
Forum Guru
Arimaa player #706
Gender:
Posts: 5928
|
|
Re: 2010 Bot Ladder
« Reply #6 on: Sep 18th, 2010, 6:15pm » |
Quote Modify
|
My first concern for the ladder is for it not to include broken and/or resource-hogging bots. We have plenty of variety available to choose from; there is no need to create unpleasant experiences from timeouts and/or server overload when plenty of well-behaved bots are available. If there are 2010 bots with no switch to restrict them to one core and no switch to limit their memory use, simply leave them off the ladder, and ask that developers put in such switches for 2011. Within that constraint, however, there should be as great a variety as possible in strategies. It is both more fun and more instructive to have different engines than multiple strengths of the same engine. My main objection to having four strengths (P1, P2, Fast, Blitz) of every engine is that the ladder gets too big. About forty bots on the ladder seems like a maximum to me, and thirty would not be too few. The ladder does not have to be exhaustive. The world will not stop turning if there is AamiraFast but no AamriaBlitz while there is a MarwinBlitz but no MarwinFast. I have a small request that is unrelated to how the ladder is constructed in general: For measuring historical rating stability it would be nice if Arimaazilla, Gnobot2005P1, Gnobot2005P1, Bomb2005P1, and Bomb2005P2 were all left in the ladder. I don't care much about the Fast and Blitz versions, since they have already varied with changing server hardware, but the fixed-performance versions allow a direct comparison to the past.
|
« Last Edit: Sep 18th, 2010, 6:19pm by Fritzlein » |
IP Logged |
|
|
|
aaaa
Forum Guru
Arimaa player #958
Posts: 768
|
|
Re: 2010 Bot Ladder
« Reply #7 on: Sep 18th, 2010, 7:04pm » |
Quote Modify
|
Ignoring time control settings or any issues with particular bots, I simply can't fathom how it is possible for there ever being too many of them on the ladder. Would people really be discouraged so much from playing any bot on the ladder if they didn't see themselves completing it in any foreseeable time, especially compared with how many bots there are on it already? Anyone starting out will have to invest a considerable time regardless. I think the ladder should have every reasonably fast bot that doesn't cause any problems to the server and doesn't have a chronic timeout problem. I count 80 such bots: bot_Arimaalon (which is oddly currently absent from the ladder, probably for being taken for the slower bot_Arimaazon), bot_ShallowBlue, bot_Arimaazilla and all other server bots with suffixes P1, P2, Fast & Blitz except for bot_Clueless2006Blitz, bot_GnoBot2009Fast and bot_GnoBot2009Blitz which time out way too often. That would be equivalent to leaving out bot_Bomb2005Lightning, bot_Arimaazon (for taking a minute per move), the three postal bots (bot_ArimaaScoreP3, bot_Bomb2005P3 and bot_Bomb2005P4), aforementioned three tardy bots and all bots with suffix CC. This would make completing the ladder a truly momentous accomplishment and lead to accurate ratings for as many bots as is reasonable. My biggest peeve with the current ladder is that Bomb is the most dominant bot on it by far and I speak from personal experience that attuning one self to just one bot invariably leads to bad habits. You can't have too much variety and there isn't much of a downside to having to play potentially similar versions of bots; if they are indeed similar, then it should likewise be no problem for players to replicate their mastery over one bot with a (near-)clone. I don't know whether one can currently advance by winning by timeout, but given the "loss by illegal move" mechanism in place and the absence of any bad apples, this should not be the case.
|
|
IP Logged |
|
|
|
Eltripas
Forum Guru
Meh-he-kah-naw
Gender:
Posts: 225
|
|
Re: 2010 Bot Ladder
« Reply #8 on: Sep 18th, 2010, 7:24pm » |
Quote Modify
|
I don't see any good reason to have limit the amount of bots in the ladder, in fact, for me, the more the merrier because I (and I'm pretty sure I'm not the only one) use the ladder as a fast way to start a game, I find a little annoying the need to go to the top rated bots to start a game.
|
|
IP Logged |
|
|
|
Hippo
Forum Guru
Arimaa player #4450
Gender:
Posts: 883
|
|
Re: 2010 Bot Ladder
« Reply #9 on: Sep 19th, 2010, 4:38am » |
Quote Modify
|
1) Bot ordering depends on their actual rankings this could explain blitz bots near one another. I would expect clueless/marwin bots on all speeds together as well. 2) It would be very time consuming to beat MarwinCC and CluelessCC especially when not done on the first try. 3) There could be possibility of assymetric time controlls that would allow to play good bots in comfortable human speed. It seems to me fast marwin/clueless with 2 minutes for human could be good training 4) I dont think lightning would be good on ladder as it is higly dependent on hardware to be able to play it.
|
« Last Edit: Sep 19th, 2010, 4:39am by Hippo » |
IP Logged |
|
|
|
aaaa
Forum Guru
Arimaa player #958
Posts: 768
|
|
Re: 2010 Bot Ladder
« Reply #10 on: Sep 19th, 2010, 10:59am » |
Quote Modify
|
The following table shows the proportion of lost rated games that were due to a timeout (plus sample size) for each server bot for which this is a positive number: bot_Clueless2006Blitz | 0.896 | 125 | bot_Clueless2010Blitz | 0.778 | 9 | bot_Rat2009Blitz | 0.667 | 3 | bot_Clueless2010Fast | 0.625 | 8 | bot_GnoBot2009Fast | 0.548 | 31 | bot_PragmaticTheory2010Fast | 0.500 | 4 | bot_Rat2009CC | 0.500 | 2 | bot_GnoBot2009Blitz | 0.452 | 31 | bot_OpFor2009CC | 0.375 | 8 | bot_Occam2004CC | 0.354 | 48 | bot_Zombie2008CC | 0.326 | 43 | bot_OpFor2008Blitz | 0.270 | 259 | bot_OpFor2008Fast | 0.244 | 246 | bot_Loc2005P1 | 0.232 | 1455 | bot_GnoBot2004CC | 0.227 | 44 | bot_Clueless2009CC | 0.217 | 23 | bot_Clueless2007Blitz | 0.187 | 235 | bot_OpFor2009Blitz | 0.179 | 28 | bot_Zombie2008Blitz | 0.176 | 301 | bot_Zombie2008Fast | 0.174 | 282 | bot_Loc2005Blitz | 0.167 | 269 | bot_Clueless2008Blitz | 0.167 | 12 | bot_Loc2005Fast | 0.159 | 226 | bot_GnoBot2006Blitz | 0.146 | 82 | bot_OpFor2009Fast | 0.145 | 55 | bot_Loc2005CC | 0.140 | 121 | bot_Marwin2010Blitz | 0.132 | 38 | bot_Loc2005P2 | 0.126 | 538 | bot_Loc2006Fast | 0.121 | 174 | bot_Bomb2005Lightning | 0.104 | 547 | bot_Sharp2010Fast | 0.100 | 10 | bot_Loc2006Blitz | 0.094 | 160 | bot_Clueless2007Fast | 0.092 | 272 | bot_GnoBot2006CC | 0.091 | 44 | bot_Clueless2006Fast | 0.090 | 155 | bot_Loc2006CC | 0.088 | 91 | bot_GnoBot2006P2 | 0.082 | 134 | bot_GnoBot2006Fast | 0.082 | 98 | bot_Clueless2009Fast | 0.079 | 152 | bot_Loc2006P2 | 0.077 | 337 | bot_Clueless2009Blitz | 0.074 | 149 | bot_Clueless2006CC | 0.071 | 56 | bot_Clueless2006P2 | 0.061 | 231 | bot_OpFor2010Blitz | 0.059 | 17 | bot_Bomb2004CC | 0.053 | 76 | bot_OpFor2008CC | 0.053 | 38 | bot_Marwin2010CC | 0.045 | 22 | bot_Loc2006P1 | 0.044 | 1507 | bot_Badger2010Blitz | 0.042 | 24 | bot_Bomb2005CC | 0.041 | 412 | bot_GnoBot2005CC | 0.039 | 103 | bot_Marwin2010Fast | 0.037 | 27 | bot_Aamira2006CC | 0.036 | 83 | bot_OpFor2008P2 | 0.034 | 589 | bot_GnoBot2005Fast | 0.034 | 414 | bot_GnoBot2005Blitz | 0.027 | 548 | bot_Loc2007Fast | 0.027 | 374 | bot_GnoBot2005P2 | 0.024 | 1270 | bot_Sharp2008CC | 0.024 | 41 | bot_Sharp2008P2 | 0.024 | 1078 | bot_Loc2007P2 | 0.024 | 927 | bot_Loc2007P1 | 0.020 | 1664 | bot_Bomb2005Fast | 0.019 | 1343 | bot_Clueless2005CC | 0.018 | 219 | bot_Bomb2005Blitz | 0.018 | 2289 | bot_Loc2007Blitz | 0.017 | 350 | bot_Clueless2005Blitz | 0.017 | 411 | bot_Clueless2006P1 | 0.014 | 694 | bot_Clueless2005Fast | 0.014 | 289 | bot_ArimaaScoreP2 | 0.013 | 3549 | bot_Arimaazilla | 0.013 | 4903 | bot_Sharp2010Blitz | 0.010 | 98 | bot_OpFor2008P1 | 0.010 | 1485 | bot_Clueless2007P1 | 0.010 | 918 | bot_Arimaazon | 0.009 | 552 | bot_Bomb2005P2 | 0.009 | 1899 | bot_GnoBot2006P1 | 0.008 | 666 | bot_Clueless2005P1 | 0.005 | 1728 | bot_Clueless2005P2 | 0.004 | 463 | bot_Aamira2006Fast | 0.004 | 517 | bot_ArimaaScoreP1 | 0.004 | 5725 | bot_Bomb2005P1 | 0.004 | 4646 | bot_Aamira2006Blitz | 0.004 | 549 | bot_ShallowBlue | 0.003 | 3009 | bot_GnoBot2005P1 | 0.003 | 2730 | bot_Clueless2007P2 | 0.003 | 367 | bot_Sharp2008P1 | 0.003 | 1529 | bot_Arimaalon | 0.001 | 2268 | bot_Aamira2006P2 | 0.001 | 1582 | bot_Aamira2006P1 | 0.001 | 2831 |
|
|
IP Logged |
|
|
|
Nombril
Forum Guru
Arimaa player #4509
Gender:
Posts: 292
|
|
Re: 2010 Bot Ladder
« Reply #11 on: Sep 19th, 2010, 3:54pm » |
Quote Modify
|
If the Bot Ladder is meant as a training tool, especially for new players, than I would emphasize variety of bots and time controls. on Sep 18th, 2010, 7:04pm, aaaa wrote:I simply can't fathom how it is possible for there ever being too many of them on the ladder. |
| Speaking as someone who just started Arimaa last fall, and used the bot ladder as a learning tool, I would not have liked a longer ladder. There were times when I would make a leap of understanding, and then suddenly the next 5 or so bots were very easy to beat. If doubling the number of bots on the ladder meant there were now 10-15 bots I have to beat in order to get to something stronger, I would have been very discouraged - out of boredom. (5 easy games was bad enough...) I'm OK with making the ladder longer if it is because we have now developed some stronger bots. But I would not want the number of bots in rough ELO bins to get too high. I think the Blitz bots were a good training tool, forcing me to learn to think/plan ahead. I also think that having one CC bot at (or near) the top of the ladder could be a good way to make the ladder taller and add variety. I certainly understand the concern of long games being a burden due to time constraints, but after having played thirty "quick" bots, would it would be possible for people to have the patience to play one CC bot?
|
|
IP Logged |
|
|
|
omar
Forum Guru
Arimaa player #2
Gender:
Posts: 1003
|
|
Re: 2010 Bot Ladder
« Reply #13 on: Sep 23rd, 2010, 6:27am » |
Quote Modify
|
on Sep 18th, 2010, 7:24pm, Eltripas wrote:I don't see any good reason to have limit the amount of bots in the ladder, in fact, for me, the more the merrier because I (and I'm pretty sure I'm not the only one) use the ladder as a fast way to start a game, I find a little annoying the need to go to the top rated bots to start a game. |
| You are right I also often use the bot ladder as a quick way to get to the available bots. I added a new 'Bots Available' option under the 'Play Now' menu. You might need to click the 'Refresh' link to see it.
|
|
IP Logged |
|
|
|
omar
Forum Guru
Arimaa player #2
Gender:
Posts: 1003
|
|
Re: 2010 Bot Ladder
« Reply #14 on: Sep 23rd, 2010, 6:31am » |
Quote Modify
|
Thanks for posting this aaaa, it will be very helpful in removing from the ladder bots that could lead to a bad experience for new players. on Sep 19th, 2010, 10:59am, aaaa wrote:The following table shows the proportion of lost rated games that were due to a timeout (plus sample size) for each server bot for which this is a positive number: bot_Clueless2006Blitz | 0.896 | 125 | bot_Clueless2010Blitz | 0.778 | 9 | bot_Rat2009Blitz | 0.667 | 3 | bot_Clueless2010Fast | 0.625 | 8 | bot_GnoBot2009Fast | 0.548 | 31 | bot_PragmaticTheory2010Fast | 0.500 | 4 | bot_Rat2009CC | 0.500 | 2 | bot_GnoBot2009Blitz | 0.452 | 31 | bot_OpFor2009CC | 0.375 | 8 | bot_Occam2004CC | 0.354 | 48 | bot_Zombie2008CC | 0.326 | 43 | bot_OpFor2008Blitz | 0.270 | 259 | bot_OpFor2008Fast | 0.244 | 246 | bot_Loc2005P1 | 0.232 | 1455 | bot_GnoBot2004CC | 0.227 | 44 | bot_Clueless2009CC | 0.217 | 23 | bot_Clueless2007Blitz | 0.187 | 235 | bot_OpFor2009Blitz | 0.179 | 28 | bot_Zombie2008Blitz | 0.176 | 301 | bot_Zombie2008Fast | 0.174 | 282 | bot_Loc2005Blitz | 0.167 | 269 | bot_Clueless2008Blitz | 0.167 | 12 | bot_Loc2005Fast | 0.159 | 226 | bot_GnoBot2006Blitz | 0.146 | 82 | bot_OpFor2009Fast | 0.145 | 55 | bot_Loc2005CC | 0.140 | 121 | bot_Marwin2010Blitz | 0.132 | 38 | bot_Loc2005P2 | 0.126 | 538 | bot_Loc2006Fast | 0.121 | 174 | bot_Bomb2005Lightning | 0.104 | 547 | bot_Sharp2010Fast | 0.100 | 10 | bot_Loc2006Blitz | 0.094 | 160 | bot_Clueless2007Fast | 0.092 | 272 | bot_GnoBot2006CC | 0.091 | 44 | bot_Clueless2006Fast | 0.090 | 155 | bot_Loc2006CC | 0.088 | 91 | bot_GnoBot2006P2 | 0.082 | 134 | bot_GnoBot2006Fast | 0.082 | 98 | bot_Clueless2009Fast | 0.079 | 152 | bot_Loc2006P2 | 0.077 | 337 | bot_Clueless2009Blitz | 0.074 | 149 | bot_Clueless2006CC | 0.071 | 56 | bot_Clueless2006P2 | 0.061 | 231 | bot_OpFor2010Blitz | 0.059 | 17 | bot_Bomb2004CC | 0.053 | 76 | bot_OpFor2008CC | 0.053 | 38 | bot_Marwin2010CC | 0.045 | 22 | bot_Loc2006P1 | 0.044 | 1507 | bot_Badger2010Blitz | 0.042 | 24 | bot_Bomb2005CC | 0.041 | 412 | bot_GnoBot2005CC | 0.039 | 103 | bot_Marwin2010Fast | 0.037 | 27 | bot_Aamira2006CC | 0.036 | 83 | bot_OpFor2008P2 | 0.034 | 589 | bot_GnoBot2005Fast | 0.034 | 414 | bot_GnoBot2005Blitz | 0.027 | 548 | bot_Loc2007Fast | 0.027 | 374 | bot_GnoBot2005P2 | 0.024 | 1270 | bot_Sharp2008CC | 0.024 | 41 | bot_Sharp2008P2 | 0.024 | 1078 | bot_Loc2007P2 | 0.024 | 927 | bot_Loc2007P1 | 0.020 | 1664 | bot_Bomb2005Fast | 0.019 | 1343 | bot_Clueless2005CC | 0.018 | 219 | bot_Bomb2005Blitz | 0.018 | 2289 | bot_Loc2007Blitz | 0.017 | 350 | bot_Clueless2005Blitz | 0.017 | 411 | bot_Clueless2006P1 | 0.014 | 694 | bot_Clueless2005Fast | 0.014 | 289 | bot_ArimaaScoreP2 | 0.013 | 3549 | bot_Arimaazilla | 0.013 | 4903 | bot_Sharp2010Blitz | 0.010 | 98 | bot_OpFor2008P1 | 0.010 | 1485 | bot_Clueless2007P1 | 0.010 | 918 | bot_Arimaazon | 0.009 | 552 | bot_Bomb2005P2 | 0.009 | 1899 | bot_GnoBot2006P1 | 0.008 | 666 | bot_Clueless2005P1 | 0.005 | 1728 | bot_Clueless2005P2 | 0.004 | 463 | bot_Aamira2006Fast | 0.004 | 517 | bot_ArimaaScoreP1 | 0.004 | 5725 | bot_Bomb2005P1 | 0.004 | 4646 | bot_Aamira2006Blitz | 0.004 | 549 | bot_ShallowBlue | 0.003 | 3009 | bot_GnoBot2005P1 | 0.003 | 2730 | bot_Clueless2007P2 | 0.003 | 367 | bot_Sharp2008P1 | 0.003 | 1529 | bot_Arimaalon | 0.001 | 2268 | bot_Aamira2006P2 | 0.001 | 1582 | bot_Aamira2006P1 | 0.001 | 2831 | |
|
|
|
IP Logged |
|
|
|
|