Arimaa Forum - Print Page


    
      
        Arimaa Forum
        (http://arimaa.com/arimaa/forum/cgi/YaBB.cgi)
      

        Arimaa >> General Discussion >> 2010 Bot Ladder
        
(Message started by: omar on Sep 18^th, 2010, 9:49am)

Title: 2010 Bot Ladder
Post by omar on Sep 18^th, 2010, 9:49am

I've been meaning to change the bots on the ladder and reset the ladder so we can all have fun clearing it again.

To help me decide what bots to put on the ladder, please post suggestions on what bots you would like to see; how many bots you think should be on the ladder, etc.

Title: Re: 2010 Bot Ladder
Post by clojure on Sep 18^th, 2010, 10:30am

I just had this discussion in the chat room. Maybe it helps?

Code:

06:00:15 clojure what i dislike strongly about the bot ladder is that in the middle you hit your head against fast/blitz bots. why not make a ladder of CC only? my progress halted totally since i don't want to play fast
06:01:06 novacat I was not aware anyone actually thought that was true, although I guess it is not surprising
06:01:33 aaaa Actually, people don't like the CC bots because the games take so long.
06:03:55 hanzack what does CC stand for?
06:04:12 novacat computer championship
06:04:28 clojure aaaa, it's pretty normal to have long games in go for example. so maybe it's different for chess players?
06:04:44 hanzack cclemon?
06:05:03 clojure i feel that the additional time has really big impact on the quality of games
06:05:19 clojure i got immersed totally different way than in blitz
06:05:48 novacat I agree, but the bot ladder does not need to reflect only that side
06:05:52 clojure and it's even more unfortunate for to watch live games when they are not slow... it's hard to know anything but very superficially
06:06:03 clojure maybe not, that's true
06:06:07 aaaa Maybe introduce 1-minute versions of the bots.
06:06:45 clojure i don't really mind what time bot takes unless i can have as much as i like
06:07:06 aaaa It should not be too easy to reach the top of the ladder. It should still represent an accomplishment. I lean against dropping the blitz bots.
06:07:10 clojure if bot plays better than me, i can learn, whether bot played at 5 sec or 2 min
06:10:25 novacat I am not good at blitz, but I did learn a lot from playing many blitz games (49 games to beat Bomb2005Blitz)
06:10:27 clojure maybe the problem is that there's a group of fast bots in sequence, so you cannot progress at all, and play them later. a better alternative could have cc/fast/blitz in every third place and thus if you always won the cc version, you got to the top but haven't finished the ladder yet
06:10:51 clojure one problem of a blitz is that of interaction with mouse is cumbersome
06:11:20 clojure the physical act of making move is too big a part of the time
06:11:46 novacat that is why there must be a plan of action
06:12:09 clojure i don't get how do you do that in a complex local situation
06:12:13 novacat knowing what moves you will start with will give you a few more seconds later
06:12:31 aaaa Well, that's how you'll develop an instinct for the tactical side of the game.
06:13:32 clojure ok, i'll have more tries then
06:13:35 novacat you also quickly learn what moves don't work, and you can get in 5 + games in an hour
06:14:12 clojure i'm not at this point so sure how important the quantity of games is for arimaa
06:14:32 clojure i rather think that the time spent alone is important
06:14:43 novacat quantity helps learn the basics, quality helps improve them
06:15:39 clojure i personally find that i don't learn much with quick play
06:16:14 novacat what I was really saying, is that if you make a bad move on move 14, you didn't just waste 30 min, but more like 6
06:16:25 clojure but sure experts in go suggest to play both blitz and slow
06:17:06 clojure if you make a bad move in slow game, it's not blunder probably
06:17:12 clojure so you'll learn something really new
06:17:43 clojure besides making bad move forces you to learn to cope as an underdog
06:18:01 clojure you are forced to make the situation complex for the opponent
06:18:32 clojure as in chess you wouldn't want to trade pieces to simplify, and in go you would make fierce tactical battles
06:20:41 novacat If you only play postal, there is plenty of time to consider situations and blitz will not help you much
06:20:43 clojure this was just some feedback on the current system. it's not a big deal
06:21:44 clojure it's funny that i actually play in postals faster than in CC
06:21:48 novacat if you play at 1 minute, it is good to trim the number of options you consider so you don't have time crunches

Is there anyone feeling the same way?

Title: Re: 2010 Bot Ladder
Post by Sconibulus on Sep 18^th, 2010, 12:55pm

I think that adding the CC bots would cause most people to never finish the ladder, not because they can't, but because they don't wish to expend the two hour time investment.

Bots playing with time controls use the full time allotment, unlike the p1 and p2 bots which move in only a few seconds, but still give you the full two minutes, to use or ignore as you see fit (I prefer ignore).

Title: Re: 2010 Bot Ladder
Post by Eltripas on Sep 18^th, 2010, 2:20pm

No CC bots on the ladder please, those games take an eternity not all the people have the time to play those, I think we should keep the bots that are on the ladder plus PT, Marwin and Badger also updating Clueless, Gnobot, Sharp and Opfor would be nice but the addition of the new bots is more important for me, I like the current time controls for p1 and p2 bots, also I think that the existence of fast and blitz bots on the ladder is ok, is not like I'm asking to put lightning bots on the ladder.

Title: Re: 2010 Bot Ladder
Post by Fritzlein on Sep 18^th, 2010, 4:07pm

on 09/18/10 at 10:30:31, clojure wrote:

Is there anyone feeling the same way?

Yes, there are people who feel the same way that you do, i.e. that playing fast and blitz bots is less fun and not the best way to improve. However, you are in the minority. We have experience to prove it. Originally there were CC bots on the ladder, and many people gave up on the ladder because of it. They only wanted games that they could finish quickly. Now that the CC bots are off the ladder, replaced by fast bots, there is a different class of people who give up on the ladder, but not as many as before.

This isn't a theoretical argument about what kind of ladder is better, just a practical observation about what the masses want. The "careful thinking" crowd is smaller than the "fast action" crowd. It is what it is.

Title: Re: 2010 Bot Ladder
Post by clojure on Sep 18^th, 2010, 5:06pm

One must notice that careful adjustation of the bot ladder gives different class of players a longer run in the ladder. This is due to the rule that three (3) of the next bots are playable. So the problem is mainly incidental: there is a sequence of fast/blitz bots in the middle of the ladder. If every 3rd bot was at least a bit slower than fast, some people might get further, and it wouldn't hinder those that don't like to waste their time on bots.

But I can fare without the ladder. I'm just saying how I feel about it.

Title: Re: 2010 Bot Ladder
Post by Fritzlein on Sep 18^th, 2010, 6:15pm

My first concern for the ladder is for it not to include broken and/or resource-hogging bots. We have plenty of variety available to choose from; there is no need to create unpleasant experiences from timeouts and/or server overload when plenty of well-behaved bots are available. If there are 2010 bots with no switch to restrict them to one core and no switch to limit their memory use, simply leave them off the ladder, and ask that developers put in such switches for 2011.

Within that constraint, however, there should be as great a variety as possible in strategies. It is both more fun and more instructive to have different engines than multiple strengths of the same engine.

My main objection to having four strengths (P1, P2, Fast, Blitz) of every engine is that the ladder gets too big. About forty bots on the ladder seems like a maximum to me, and thirty would not be too few. The ladder does not have to be exhaustive. The world will not stop turning if there is AamiraFast but no AamriaBlitz while there is a MarwinBlitz but no MarwinFast.

I have a small request that is unrelated to how the ladder is constructed in general: For measuring historical rating stability it would be nice if Arimaazilla, Gnobot2005P1, Gnobot2005P1, Bomb2005P1, and Bomb2005P2 were all left in the ladder. I don't care much about the Fast and Blitz versions, since they have already varied with changing server hardware, but the fixed-performance versions allow a direct comparison to the past.

Title: Re: 2010 Bot Ladder
Post by aaaa on Sep 18^th, 2010, 7:04pm

Ignoring time control settings or any issues with particular bots, I simply can't fathom how it is possible for there ever being too many of them on the ladder. Would people really be discouraged so much from playing any bot on the ladder if they didn't see themselves completing it in any foreseeable time, especially compared with how many bots there are on it already? Anyone starting out will have to invest a considerable time regardless.

I think the ladder should have every reasonably fast bot that doesn't cause any problems to the server and doesn't have a chronic timeout problem.
I count 80 such bots: bot_Arimaalon (which is oddly currently absent from the ladder, probably for being taken for the slower bot_Arimaazon), bot_ShallowBlue, bot_Arimaazilla and all other server bots with suffixes P1, P2, Fast & Blitz except for bot_Clueless2006Blitz, bot_GnoBot2009Fast and bot_GnoBot2009Blitz which time out way too often.
That would be equivalent to leaving out bot_Bomb2005Lightning, bot_Arimaazon (for taking a minute per move), the three postal bots (bot_ArimaaScoreP3, bot_Bomb2005P3 and bot_Bomb2005P4), aforementioned three tardy bots and all bots with suffix CC.
This would make completing the ladder a truly momentous accomplishment and lead to accurate ratings for as many bots as is reasonable.

My biggest peeve with the current ladder is that Bomb is the most dominant bot on it by far and I speak from personal experience that attuning one self to just one bot invariably leads to bad habits. You can't have too much variety and there isn't much of a downside to having to play potentially similar versions of bots; if they are indeed similar, then it should likewise be no problem for players to replicate their mastery over one bot with a (near-)clone.

I don't know whether one can currently advance by winning by timeout, but given the "loss by illegal move" mechanism in place and the absence of any bad apples, this should not be the case.

Title: Re: 2010 Bot Ladder
Post by Eltripas on Sep 18^th, 2010, 7:24pm

I don't see any good reason to have limit the amount of bots in the ladder, in fact, for me, the more the merrier because I (and I'm pretty sure I'm not the only one) use the ladder as a fast way to start a game, I find a little annoying the need to go to the top rated bots to start a game.

Title: Re: 2010 Bot Ladder
Post by Hippo on Sep 19^th, 2010, 4:38am

1) Bot ordering depends on their actual rankings this could explain blitz bots near one another. I would expect clueless/marwin bots on all speeds together as well.

2) It would be very time consuming to beat MarwinCC and CluelessCC especially when not done on the first try.

3) There could be possibility of assymetric time controlls that would allow to play good bots in comfortable human speed. It seems to me fast marwin/clueless with 2 minutes for human could be good training

4) I dont think lightning would be good on ladder as it is higly dependent on hardware to be able to play it.

Title: Re: 2010 Bot Ladder
Post by aaaa on Sep 19^th, 2010, 10:59am

The following table shows the proportion of lost rated games that were due to a timeout (plus sample size) for each server bot for which this is a positive number:

bot_Clueless2006Blitz	0.896	125
bot_Clueless2010Blitz	0.778	9
bot_Rat2009Blitz	0.667	3
bot_Clueless2010Fast	0.625	8
bot_GnoBot2009Fast	0.548	31
bot_PragmaticTheory2010Fast	0.500	4
bot_Rat2009CC	0.500	2
bot_GnoBot2009Blitz	0.452	31
bot_OpFor2009CC	0.375	8
bot_Occam2004CC	0.354	48
bot_Zombie2008CC	0.326	43
bot_OpFor2008Blitz	0.270	259
bot_OpFor2008Fast	0.244	246
bot_Loc2005P1	0.232	1455
bot_GnoBot2004CC	0.227	44
bot_Clueless2009CC	0.217	23
bot_Clueless2007Blitz	0.187	235
bot_OpFor2009Blitz	0.179	28
bot_Zombie2008Blitz	0.176	301
bot_Zombie2008Fast	0.174	282
bot_Loc2005Blitz	0.167	269
bot_Clueless2008Blitz	0.167	12
bot_Loc2005Fast	0.159	226
bot_GnoBot2006Blitz	0.146	82
bot_OpFor2009Fast	0.145	55
bot_Loc2005CC	0.140	121
bot_Marwin2010Blitz	0.132	38
bot_Loc2005P2	0.126	538
bot_Loc2006Fast	0.121	174
bot_Bomb2005Lightning	0.104	547
bot_Sharp2010Fast	0.100	10
bot_Loc2006Blitz	0.094	160
bot_Clueless2007Fast	0.092	272
bot_GnoBot2006CC	0.091	44
bot_Clueless2006Fast	0.090	155
bot_Loc2006CC	0.088	91
bot_GnoBot2006P2	0.082	134
bot_GnoBot2006Fast	0.082	98
bot_Clueless2009Fast	0.079	152
bot_Loc2006P2	0.077	337
bot_Clueless2009Blitz	0.074	149
bot_Clueless2006CC	0.071	56
bot_Clueless2006P2	0.061	231
bot_OpFor2010Blitz	0.059	17
bot_Bomb2004CC	0.053	76
bot_OpFor2008CC	0.053	38
bot_Marwin2010CC	0.045	22
bot_Loc2006P1	0.044	1507
bot_Badger2010Blitz	0.042	24
bot_Bomb2005CC	0.041	412
bot_GnoBot2005CC	0.039	103
bot_Marwin2010Fast	0.037	27
bot_Aamira2006CC	0.036	83
bot_OpFor2008P2	0.034	589
bot_GnoBot2005Fast	0.034	414
bot_GnoBot2005Blitz	0.027	548
bot_Loc2007Fast	0.027	374
bot_GnoBot2005P2	0.024	1270
bot_Sharp2008CC	0.024	41
bot_Sharp2008P2	0.024	1078
bot_Loc2007P2	0.024	927
bot_Loc2007P1	0.020	1664
bot_Bomb2005Fast	0.019	1343
bot_Clueless2005CC	0.018	219
bot_Bomb2005Blitz	0.018	2289
bot_Loc2007Blitz	0.017	350
bot_Clueless2005Blitz	0.017	411
bot_Clueless2006P1	0.014	694
bot_Clueless2005Fast	0.014	289
bot_ArimaaScoreP2	0.013	3549
bot_Arimaazilla	0.013	4903
bot_Sharp2010Blitz	0.010	98
bot_OpFor2008P1	0.010	1485
bot_Clueless2007P1	0.010	918
bot_Arimaazon	0.009	552
bot_Bomb2005P2	0.009	1899
bot_GnoBot2006P1	0.008	666
bot_Clueless2005P1	0.005	1728
bot_Clueless2005P2	0.004	463
bot_Aamira2006Fast	0.004	517
bot_ArimaaScoreP1	0.004	5725
bot_Bomb2005P1	0.004	4646
bot_Aamira2006Blitz	0.004	549
bot_ShallowBlue	0.003	3009
bot_GnoBot2005P1	0.003	2730
bot_Clueless2007P2	0.003	367
bot_Sharp2008P1	0.003	1529
bot_Arimaalon	0.001	2268
bot_Aamira2006P2	0.001	1582
bot_Aamira2006P1	0.001	2831

Title: Re: 2010 Bot Ladder
Post by Nombril on Sep 19^th, 2010, 3:54pm

If the Bot Ladder is meant as a training tool, especially for new players, than I would emphasize variety of bots and time controls.

on 09/18/10 at 19:04:51, aaaa wrote:

I simply can't fathom how it is possible for there ever being too many of them on the ladder.

Speaking as someone who just started Arimaa last fall, and used the bot ladder as a learning tool, I would not have liked a longer ladder. There were times when I would make a leap of understanding, and then suddenly the next 5 or so bots were very easy to beat. If doubling the number of bots on the ladder meant there were now 10-15 bots I have to beat in order to get to something stronger, I would have been very discouraged - out of boredom. (5 easy games was bad enough...) I'm OK with making the ladder longer if it is because we have now developed some stronger bots. But I would not want the number of bots in rough ELO bins to get too high.

I think the Blitz bots were a good training tool, forcing me to learn to think/plan ahead.

I also think that having one CC bot at (or near) the top of the ladder could be a good way to make the ladder taller and add variety. I certainly understand the concern of long games being a burden due to time constraints, but after having played thirty "quick" bots, would it would be possible for people to have the patience to play one CC bot?

Title: Re: 2010 Bot Ladder
Post by clojure on Sep 19^th, 2010, 5:20pm

So, as encouraged to play the ladder with blitz bots, I gave a shot: http://arimaa.com/arimaa/gameroom/opengamewin.cgi?client=1&gameid=154975&role=v&side=b

See the move 28g by bot. ???

Is it normal that the game ends as half of the board nearly empty and all the pieces in congestion in the other?

Title: Re: 2010 Bot Ladder
Post by omar on Sep 23^rd, 2010, 6:27am

on 09/18/10 at 19:24:23, Eltripas wrote:

You are right I also often use the bot ladder as a quick way to get to the available bots. I added a new 'Bots Available' option under the 'Play Now' menu. You might need to click the 'Refresh' link to see it.

Title: Re: 2010 Bot Ladder
Post by omar on Sep 23^rd, 2010, 6:31am

Thanks for posting this aaaa, it will be very helpful in removing from the ladder bots that could lead to a bad experience for new players.

on 09/19/10 at 10:59:55, aaaa wrote:

The following table shows the proportion of lost rated games that were due to a timeout (plus sample size) for each server bot for which this is a positive number:

bot_Clueless2006Blitz	0.896	125
bot_Clueless2010Blitz	0.778	9
bot_Rat2009Blitz	0.667	3
bot_Clueless2010Fast	0.625	8
bot_GnoBot2009Fast	0.548	31
bot_PragmaticTheory2010Fast	0.500	4
bot_Rat2009CC	0.500	2
bot_GnoBot2009Blitz	0.452	31
bot_OpFor2009CC	0.375	8
bot_Occam2004CC	0.354	48
bot_Zombie2008CC	0.326	43
bot_OpFor2008Blitz	0.270	259
bot_OpFor2008Fast	0.244	246
bot_Loc2005P1	0.232	1455
bot_GnoBot2004CC	0.227	44
bot_Clueless2009CC	0.217	23
bot_Clueless2007Blitz	0.187	235
bot_OpFor2009Blitz	0.179	28
bot_Zombie2008Blitz	0.176	301
bot_Zombie2008Fast	0.174	282
bot_Loc2005Blitz	0.167	269
bot_Clueless2008Blitz	0.167	12
bot_Loc2005Fast	0.159	226
bot_GnoBot2006Blitz	0.146	82
bot_OpFor2009Fast	0.145	55
bot_Loc2005CC	0.140	121
bot_Marwin2010Blitz	0.132	38
bot_Loc2005P2	0.126	538
bot_Loc2006Fast	0.121	174
bot_Bomb2005Lightning	0.104	547
bot_Sharp2010Fast	0.100	10
bot_Loc2006Blitz	0.094	160
bot_Clueless2007Fast	0.092	272
bot_GnoBot2006CC	0.091	44
bot_Clueless2006Fast	0.090	155
bot_Loc2006CC	0.088	91
bot_GnoBot2006P2	0.082	134
bot_GnoBot2006Fast	0.082	98
bot_Clueless2009Fast	0.079	152
bot_Loc2006P2	0.077	337
bot_Clueless2009Blitz	0.074	149
bot_Clueless2006CC	0.071	56
bot_Clueless2006P2	0.061	231
bot_OpFor2010Blitz	0.059	17
bot_Bomb2004CC	0.053	76
bot_OpFor2008CC	0.053	38
bot_Marwin2010CC	0.045	22
bot_Loc2006P1	0.044	1507
bot_Badger2010Blitz	0.042	24
bot_Bomb2005CC	0.041	412
bot_GnoBot2005CC	0.039	103
bot_Marwin2010Fast	0.037	27
bot_Aamira2006CC	0.036	83
bot_OpFor2008P2	0.034	589
bot_GnoBot2005Fast	0.034	414
bot_GnoBot2005Blitz	0.027	548
bot_Loc2007Fast	0.027	374
bot_GnoBot2005P2	0.024	1270
bot_Sharp2008CC	0.024	41
bot_Sharp2008P2	0.024	1078
bot_Loc2007P2	0.024	927
bot_Loc2007P1	0.020	1664
bot_Bomb2005Fast	0.019	1343
bot_Clueless2005CC	0.018	219
bot_Bomb2005Blitz	0.018	2289
bot_Loc2007Blitz	0.017	350
bot_Clueless2005Blitz	0.017	411
bot_Clueless2006P1	0.014	694
bot_Clueless2005Fast	0.014	289
bot_ArimaaScoreP2	0.013	3549
bot_Arimaazilla	0.013	4903
bot_Sharp2010Blitz	0.010	98
bot_OpFor2008P1	0.010	1485
bot_Clueless2007P1	0.010	918
bot_Arimaazon	0.009	552
bot_Bomb2005P2	0.009	1899
bot_GnoBot2006P1	0.008	666
bot_Clueless2005P1	0.005	1728
bot_Clueless2005P2	0.004	463
bot_Aamira2006Fast	0.004	517
bot_ArimaaScoreP1	0.004	5725
bot_Bomb2005P1	0.004	4646
bot_Aamira2006Blitz	0.004	549
bot_ShallowBlue	0.003	3009
bot_GnoBot2005P1	0.003	2730
bot_Clueless2007P2	0.003	367
bot_Sharp2008P1	0.003	1529
bot_Arimaalon	0.001	2268
bot_Aamira2006P2	0.001	1582
bot_Aamira2006P1	0.001	2831

Title: Re: 2010 Bot Ladder
Post by omar on Sep 23^rd, 2010, 8:52am

I made a graph of the ratings of the bots on the ladder. The ratings can fluctuate around as bots are played, but the lower rated bots have been played a lot and have fairly stable ratings. It seems that the bot ladder is not a gradually increasing ramp, but has several steep steps followed by flat plateaus. This could explain the kind of experience Nombril described.

http://i55.tinypic.com/xt99s.jpg

Title: Re: 2010 Bot Ladder
Post by rbarreira on Sep 23^rd, 2010, 9:04am

Marwin 2010 blitz has played a lot of games and it's managing to hold on to a near 2200 rating, so that's one more reason why it should probably be added to the ladder.

The other reason is that it's a new bot. it's good to show that the state of the art in bots is improving, someone might get the wrong idea by seeing bomb2005Fast at the top of the ladder.

Title: Re: 2010 Bot Ladder
Post by omar on Sep 23^rd, 2010, 4:04pm

Thanks for the feedback on this everyone. It seems like the bot ladder is serving two different functions. For the new players it's a way to learn Arimaa while trying to reach the top of the ladder. For the established players it's a way to easily find the bots and keep track of which bots you've defeated and which new bots you have yet to defeat. If the ladder is to serve the new players it would not be good to put all the available bots on the ladder. The ladder would seem too long and discouraging. If the ladder is to serve the established player it does make sense to put all the bots on it and allow the ladder to grow each year. So I've decided to create two separate ladders.

For the new players we will continue to use the existing ladder (with some changes of which bots appear):
http://arimaa.com/arimaa/gameroom/botLadder.cgi

While the established players can start using this ladder:
http://arimaa.com/arimaa/gameroom/botLadderAll.cgi

which shows all the available bots sorted by your winning percentage and the bots rating. I've linked this under the 'Play Now' menu as 'Bots Available' so it can easily be accessed. You can also pass a parameter to this script to view how the ladder looks for other players (for example: ?u=omar)

On the new player ladder I want to make sure that all the bots there move fast and do not time out or send illegal moves. Also I am considering adding some more easy bots that we create just for new players to fill in the big jumps in ratings of the bots. These new bots would be stronger than ArimaaScoreP1 but not too strong and provide a variety of different strategies. They would get progressively stronger and make the ladder ratings more evenly distributed. I am also considering giving all the bots on the new player ladder names which don't have years in them and rather using names similar to ShallowBlue and Arimaazilla. This is so that the new player ladder can remain static and always have the same set of bots on it year after year. The bots on this ladder will provide a way for us to measure how new players progress in Arimaa. I am also considering making all the bots on the new player ladder fixed strength and varying the amount of time the human player gets to increase the difficulty level. I'm sure if I played Marwin2010P2, Clueless2010P2 or the other latest bots at blitz speed I would still find them very difficult to beat. I think about 30 bots on the ladder should be low enough where the ladder does not seem too difficult to finish and has enough bots so that the rating gap between them will not be too large. If the bots span from a rating of 1000 to 2000 then there would be about 33 rating points between two near bots.

Let me know your thoughts on this. Thanks.

Title: Re: 2010 Bot Ladder
Post by qswanger on Sep 23^rd, 2010, 4:35pm

The 'Play' links from this page do not appear to be working. They all open up the bot_ArimaaScoreP1 Control Page

Title: Re: 2010 Bot Ladder
Post by Eltripas on Sep 23^rd, 2010, 11:38pm

I like the new ladder but I'd like to have Marwin on the old ladder too.

Title: Re: 2010 Bot Ladder
Post by omar on Sep 24^th, 2010, 12:24am

on 09/23/10 at 16:35:50, qswanger wrote:

The 'Play' links from this page do not appear to be working. They all open up the bot_ArimaaScoreP1 Control Page

Yes, I just noticed that too. It's fixed now.

Title: Re: 2010 Bot Ladder
Post by Hirocon on Sep 24^th, 2010, 1:00am

The stats on the new ladder say I've played two games against Bot_Marwin2010Blitz, but I've never played that bot (or any Marwin bot for that matter).

Title: Re: 2010 Bot Ladder
Post by Tuks on Sep 24^th, 2010, 1:17am

can you make the "people who completed the ladder" for both ladders?

it would be cool if we could have achievements, like:

people who completed the ladder
people who beat all fast bots 50% +
people who beat all fast bots 90% +
etc.

Title: Re: 2010 Bot Ladder
Post by omar on Sep 24^th, 2010, 3:04pm

on 09/24/10 at 01:00:54, Hirocon wrote:

The stats on the new ladder say I've played two games against Bot_Marwin2010Blitz, but I've never played that bot (or any Marwin bot for that matter).

Thanks for finding this bug. Should be fixed now.

Title: Re: 2010 Bot Ladder
Post by omar on Sep 24^th, 2010, 3:12pm

on 09/24/10 at 01:17:50, Tuks wrote:

Yes, I was planning to make a list of people who completed the advanced ladder based on having won at least one game against all the bots. Making lists based on finer points like "beat all fast bots 50%+" would probably not be as easy. But, I'll look into it when I work on that.

Title: Re: 2010 Bot Ladder
Post by Tuks on Sep 25^th, 2010, 5:12am

thanks omar,

there are also a couple bots on the list that cant be played like all the P3 P4 bots

Title: Re: 2010 Bot Ladder
Post by omar on Sep 25^th, 2010, 9:02am

on 09/25/10 at 05:12:51, Tuks wrote:

thanks omar,

there are also a couple bots on the list that cant be played like all the P3 P4 bots

Yes, we used those to experiment with how long Bomb would take to do a full P3 or P4 search and it was way too long for interactive play so we tried some postal games with it. I've taken out of the list now. Thanks.

Title: Re: 2010 Bot Ladder
Post by aaaa on Sep 25^th, 2010, 2:30pm

Some more non-working bots: bot_Badger2009CC, bot_GnoBot2009Fast, bot_GnoBot2009Blitz and bot_GnoBot2009CC. Every one of these will send an illegal first move, except for bot_GnoBot2009CC, for which there is no starting permission (but which would probably malfunction likewise).

Title: Re: 2010 Bot Ladder
Post by clojure on Sep 25^th, 2010, 5:49pm

This is lame to say but in the bot ladder there's link "how 2 win". Would it be possible to change to something more informative or at least to "How to win" ;)

Title: Re: 2010 Bot Ladder
Post by omar on Sep 28^th, 2010, 6:25am

on 09/25/10 at 14:30:55, aaaa wrote:

Thanks. I took Badger off the list and installed some missing libraries to get GnoBot2009 working again. The CC version works also, but takes up 60% of the system RAM and could cause the server to hang if other bots were running. I'll take it off the list.

Title: Re: 2010 Bot Ladder
Post by omar on Sep 28^th, 2010, 6:50am

on 09/25/10 at 17:49:35, clojure wrote:

This is lame to say but in the bot ladder there's link "how 2 win". Would it be possible to change to something more informative or at least to "How to win" ;)

Thanks. Changed this also :-)

Title: Re: 2010 Bot Ladder
Post by Tuks on Oct 3^rd, 2010, 2:25pm

hey, when you going to someone else's profile and click their original ladder it shows who they have beaten and who they havent, it would be cool if that was also the case for the full bot list (the advanced ladder so to say)

Title: Re: 2010 Bot Ladder
Post by ocmiente on Oct 6^th, 2010, 8:27pm

bot_Clueless2006P1 doesn't appear to be working.
bot_Clueless2006P2 also doesn't appear to be working.
I think that other versions of bot_Clueless aren't working either. I've tried to launch some of them over the past three days, with no luck.

Title: Re: 2010 Bot Ladder
Post by omar on Oct 7^th, 2010, 12:58am

@Tuks, good suggestion. Added link to the Advanced ladder on the users profile page.

@ocmiente, thanks for finding this. All the Clueless2006 bots broke when I disabled the CC version. It's fixed now.

Title: Re: 2010 Bot Ladder
Post by Nombril on Oct 7^th, 2010, 11:40pm

bot_Clueless2006P1 and P2 both timed out after 3 moves tonight.

Title: Re: 2010 Bot Ladder
Post by Fritzlein on Oct 7^th, 2010, 11:51pm

I finally got a chance to look at the new bot ladder. I like comprehensive aspect of including "all the bots fit to play", while still preserving a subset of that as the standard, recommended challenge for new players. Thanks, Omar, for adding this feature. The rating graph of bots on the standard ladder is good argument for adding a few more on the low end and weeding out a few in the middle.

I also like being able to look at other players' comprehensive bot results in sorted order. One of the more interesting ones is ArifSyed's (http://arimaa.com/arimaa/gameroom/botLadderAll.cgi?u=7411). He has scored +1878 against bot_Bomb2005P1 (1923 wins and 45 losses). I'm going to go out on a limb and say this is the highest plus score of any player over any other in the history of the Arimaa server. ;) Also his 339-0 margin over Arimaazilla is probably the biggest perfect score.

Title: Re: 2010 Bot Ladder
Post by ocmiente on Oct 8^th, 2010, 3:21am

on 10/07/10 at 23:40:05, Nombril wrote:

bot_Clueless2006P1 and P2 both timed out after 3 moves tonight.

I experienced these timeouts tonight:
bot_Clueless2006P1: 3 moves
bot_Clueless2006P2: 3 moves
bot_Clueless2006Fast: 3 moves
bot_Clueless2006Blitz: 5 moves

Title: Re: 2010 Bot Ladder
Post by omar on Oct 8^th, 2010, 3:26pm

on 10/07/10 at 23:40:05, Nombril wrote:

bot_Clueless2006P1 and P2 both timed out after 3 moves tonight.

It was crashing with an out of memory error. Increased the memory it can use from 0.5GB to 0.7GB and now it seems to be working OK. Although when I ran bot_Clueless2006P1 while other users were playing bot_OpFor2008P2 and bot_Sharp2008P1 it crashed the system.

I think we will have to start disabling the bots while event games are going on.

Title: Re: 2010 Bot Ladder
Post by omar on Oct 8^th, 2010, 7:58pm

on 09/23/10 at 16:04:00, omar wrote:

Also I am considering adding some more easy bots that we create just for new players to fill in the big jumps in ratings of the bots. These new bots would be stronger than ArimaaScoreP1 but not too strong and provide a variety of different strategies. They would get progressively stronger and make the ladder ratings more evenly distributed.

Would anyone like to take a crack at creating some of these simple bots. It would be good if they are written in Java. I think that will allow the bots to continue running for a long time even as the hardware changes. Jeff has contributed a sample bot written in Java which would be a good starting point for this. It is available on the downloads page: http://arimaa.com/arimaa/download/

Title: Re: 2010 Bot Ladder
Post by clojure on Oct 9^th, 2010, 1:00pm

on 10/08/10 at 19:58:13, omar wrote:

Would anyone like to take a crack at creating some of these simple bots. It would be good if they are written in Java.

I'll try to keep this in my mind so that when I begin developing my participant for the challenge, I progressively make these simple bots, too. The language will not be Java but JDK based anyway (Scala, Clojure). Depends how I will feel about D language (its maturity is under question).

Don't rely on me though :-[

Title: Re: 2010 Bot Ladder
Post by Tuks on Nov 6^th, 2010, 12:07pm

can you make the ladder completed page for the advanced ladder please :)

i have a suspicion that there is actually no one who has beaten every single working bot

Title: Re: 2010 Bot Ladder
Post by omar on Nov 9^th, 2010, 11:10pm

Thanks for the reminder Tuks. I'll try to do it this weekend.

I think you are right; there probably aren't many players who have defeated let alone played all the bots yet. I know I haven't.

Title: Re: 2010 Bot Ladder
Post by omar on Nov 21^st, 2010, 5:44pm

If you view your Advanced Ladder there is now a link for 'Advanced Ladder Climbers'. You can click on that to see who has completed the Advanced Ladder. As Tuks suspected, currently there is no one who defeated all the available bots yet.

Title: Re: 2010 Bot Ladder
Post by ocmiente on Nov 29^th, 2010, 1:52pm

Looks like 'settler' is the first to complete the ladder, congrats!

I was watching settler move up the ladder, and thought that marwin would stop settler, since it at first appeared that marwin was immune to bait and tackle (http://arimaa.com/arimaa/mwiki/index.php?title=Bot_Slaying&redirect=no#Bait_and_Tackle). settler put on an impressive display (http://arimaa.com/arimaa/games/jsShowGame.cgi?gid=161974&s=w) of baiting marwin in the end, proving that even marwin is not immune to that strategy, sacrificing two rabbits at the beginning of the game to get marwin to take the bait.

settler not only beat marwin blitz in the final ladder game (http://arimaa.com/arimaa/games/jsShowGame.cgi?gid=162101&s=b) (with a slightly different strategy than bait and tackle, but still focused on an elephant blockade), settler thrashed the bot, eliminating all of marwin's pieces without losing any of settler's own.

Title: Re: 2010 Bot Ladder
Post by Fritzlein on Nov 29^th, 2010, 8:22pm

on 11/29/10 at 13:52:21, ocmiente wrote:

Looks like 'settler' is the first to complete the ladder, congrats!

Hmmm, but it still says here (http://arimaa.com/arimaa/gameroom/botLadderAllComp.cgi) that no one has completed the advanced ladder.

Title: Re: 2010 Bot Ladder
Post by aaaa on Nov 29^th, 2010, 8:52pm

Look at the date of update.

Title: Re: 2010 Bot Ladder
Post by aaaa on Nov 30^th, 2010, 10:31am

Now, it still not showing settler is a legitimate issue. Maybe it's because not all victory conditions count.

Title: Re: 2010 Bot Ladder
Post by ocmiente on Nov 30^th, 2010, 12:25pm

on 11/30/10 at 10:31:14, aaaa wrote:

Now, it still not showing settler is a legitimate issue. Maybe it's because not all victory conditions count.

I hope that's not the case. If it is, I'd like to know what the conditions are that count.

settler has definitely cleared the ladder, as far as I can tell by looking at settler's ladder page (http://arimaa.com/arimaa/gameroom/botLadderAll.cgi?u=13299).

Title: Re: 2010 Bot Ladder
Post by Eltripas on Nov 30^th, 2010, 6:51pm

on 11/29/10 at 13:52:21, ocmiente wrote:

I was watching settler move up the ladder, and thought that marwin would stop settler, since it at first appeared that marwin was immune to bait and tackle (http://arimaa.com/arimaa/mwiki/index.php?title=Bot_Slaying&redirect=no#Bait_and_Tackle). settler put on an impressive display (http://arimaa.com/arimaa/games/jsShowGame.cgi?gid=161974&s=w) of baiting marwin in the end, proving that even marwin is not immune to that strategy, sacrificing two rabbits at the beginning of the game to get marwin to take the bait.

That was epic

Title: Re: 2010 Bot Ladder
Post by rabbits on Nov 30^th, 2010, 9:15pm

on 11/30/10 at 18:51:51, Eltripas wrote:

That was epic

Agreed! Thanks for linking those games Ocmiente!

Title: Re: 2010 Bot Ladder
Post by omar on Dec 2^nd, 2010, 11:25am

on 11/29/10 at 20:22:09, Fritzlein wrote:

Hmmm, but it still says here (http://arimaa.com/arimaa/gameroom/botLadderAllComp.cgi) that no one has completed the advanced ladder.

Case sensitivity bug :-) Should be fixed now.

Title: Re: 2010 Bot Ladder
Post by omar on Dec 2^nd, 2010, 11:28am

on 11/29/10 at 13:52:21, ocmiente wrote:

Congrats settler for being the first to clear the Advanced ladder. Those are some amazing wins against Marwin.

Title: Re: 2010 Bot Ladder
Post by Belteshazzar on Jan 4^th, 2011, 12:43am

In regards to the beginning of this thread, I would think that a revamped bot ladder would be a good idea. The fact is, very few people are going to put in the time to complete the "Advanced Bot Ladder" even though some are obviously capable of it, and the normal bot ladder is out of date.

Title: Re: 2010 Bot Ladder
Post by omar on Jan 6^th, 2011, 9:15am

I'll be changing the 'new player' bot ladder, but mainly to make it a better experience for new players. I don't think this ladder will be interesting to established players; it is not intended to be. It can't serve two purposes (see my posting of Sep 23). If the Advanced ladder takes too long to complete (it will get longer each year as the latest bots are added), maybe we need an intermediate ladder. I'll consider that after the 2011 events.

Title: Re: 2010 Bot Ladder
Post by Fritzlein on Jan 6^th, 2011, 9:48am

on 01/06/11 at 09:15:11, omar wrote:

I think that's a keen realization. The bot ladder should be for getting newcomers hooked and for giving them a good experience. That means fast-playing opponents, a variety of styles, opponents that don't time out or make illegal moves (since these are very unsatisfying ways to win), and a steady progression in the challenge.

For the server's sake we don't want any commonly-played bot to be a resource hog. And I will take the opportunity to repeat my request (insofar as it doesn't interfere with other objectives for the ladder ;D): For measuring historical rating stability it would be nice if Arimaazilla, Gnobot2005P1, Gnobot2005P2, Bomb2005P1, and Bomb2005P2 were all left in the ladder. That's the simplest way to track game room rating inflation/deflation.

Title: Re: 2010 Bot Ladder
Post by Fritzlein on Jan 6^th, 2011, 9:54am

I'm suddenly struck by the fact that two out of the three people who have completed the full advanced bot ladder have not signed up for the World Championship. That makes me wonder whether some day the World Champion will no longer be the best player to defend the Arimaa Challege. Will the Arimaa community develop bot-bashing specialists who couldn't beat the human World Champion but who could do better than the World Champion could at defeating the Computer Champion?

I hope not, because if that happens it will say something negative about the value of strategic understanding in Arimaa.

Title: Re: 2010 Bot Ladder
Post by Tuks on Jan 6^th, 2011, 10:42am

thats mainly because of the cc bots, i know i finished off all the other bots a long time ago but didn't really have the time to play the 10 cc bots i needed to complete the ladder,

i'm not complaining, i think the requirement to beat cc bots as well only makes the achievement of beating the ladder more rewarding, its not just about having the skill to win, but also the determination to get through them all

i think its too early to worry about that concern, at least now i am pretty sure you, c&g and adanac are still our best defenders

Title: Re: 2010 Bot Ladder
Post by Nombril on Jan 6^th, 2011, 1:16pm

Fritz - I think you would need to check if the 'bot-bashing' specialists are able to beat a new bot on their first few tries. It often seems that it takes some experimentation to find the right technique to bash each bot. Since the defender plays the Computer Champion without a chance to practice, many of the techniques wouldn't be safe to rely on. For example, settler found that Marwin (Blitz 2010) was willing to fall for a Bait and Tackle. But he has only won 2 out of 27 games against it. (I didn't check the other speeds...)

Title: Re: 2010 Bot Ladder
Post by tize on Jan 11^th, 2011, 4:47pm

I think that in all unlearning bots there will always be special strategys that can be developed to easier beat that bot. The longer the bot is static the more this strategys can be perfected. And these strategys might not have much to do with normal human arimaa strategy, but more with oversimplification, bugs or a developers wrong understanding of the game.

Bomb is a good example of a static and well studied bot that have a few easy to learn strategys to follow to be able to beat it "before" one is stronger for real.

So it could be that in a distant(?) future the top player will have a harder time winning than a weaker player that is more trained in the flaws of the bot.

But defending the challenge with not the best players but with the player with the best bot flaw understanding is a much riskier way, in my opinion. Because that flaw could possibly be fix by a single simple bug fix.

As with settlers record against blitz marwin he has won both of his games in the last nine games, which could indicate that he is starting to find a working strategy... Let's hope it's just a single simple bug to fix... :)

Title: Re: 2010 Bot Ladder
Post by Tuks on Jul 3^rd, 2011, 8:58am

hey omar, when are the new bots going to be up and running, i just finished exams and i need a goal

Title: Re: 2010 Bot Ladder
Post by arimaaphile on Jul 24^th, 2011, 10:38pm

Friends,

Do I miss something? I won bots somewhat easily up to 1510 but now I have difficulty. No matter what I do I lose to bots. Granted that I make lots of blunders but I'm suspicious I miss something. Maybe there is something I don't get. I noticed that bot's elephant always pulls my rabbits from home ranks and I have no idea what to do against that.
Any advice is appreciated.

Title: Re: 2010 Bot Ladder
Post by Fritzlein on Jul 24^th, 2011, 11:09pm

Have you clicked on the "How to win" links on the ladder next to the bots you are trying to beat? Those link to recent games where the bot has lost. If you click on some games you can replay them to see how other people are exploiting bot weaknesses. That might give you some ideas of different plans of attack.

Title: Re: 2010 Bot Ladder
Post by ginrunner on Jul 25^th, 2011, 3:10am

i noticed when i was playing them I would plateau for a bit until i learned how to prevent what was happening to me and once I figured that out I would soar to a new plateau. Just keep at it.

Title: Re: 2010 Bot Ladder
Post by arimaa_master on Jul 25^th, 2011, 8:32am

on 07/24/11 at 22:38:56, arimaaphile wrote:

Yes, I felt that hopeless few years ago around bots rated 1800 - especially fast and quicker bots.

I got through only after playing few dozens of games against them. Believe me - you will learn with every lost game 10 times more than with won game (if you take it seriously searching for ideas what went wrong).

Title: Re: 2010 Bot Ladder
Post by UruramTururam on Jul 25^th, 2011, 8:59am

on 07/24/11 at 22:38:56, arimaaphile wrote:

I won bots somewhat easily up to 1510 but now I have difficulty.

I'd add: this it the threshold where intuitive playing is no longer sufficient and you have to start learning tactics, and - last but not least - start avoid easy mistakes that mid-level bts know how to exploit.