Arimaa Forum - Print Page


    
      
        Arimaa Forum
        (http://arimaa.com/arimaa/forum/cgi/YaBB.cgi)
      

        Arimaa >> Events >> State of the Challenge 2012
        
(Message started by: Fritzlein on Jan 16^th, 2012, 8:04pm)

Title: State of the Challenge 2012
Post by Fritzlein on Jan 16^th, 2012, 8:04pm

It has been a while since we have had a general discussion of the state of the Arimaa Challenge, where everyone pitches in with perceptions of the current man/machine balance and prognostications for when computers will surpass us. Here are some predictions from 2009 (http://arimaa.com/arimaa/forum/cgi/YaBB.cgi?board=events;action=display;num=1249656584;start=0#0)

My own optimism that humans would defend the Challenge through 2020 is gradually eroding. I now put the chance of Omar having to pay out his $10,000 prize at 30%. Sharp and marwin each had performance ratings a bit over 2100 in last year's qualifying, and I expect the top bot to reach 2200 this year. Sharp might reach that level purely from parallelization; it won the Computer Championship last year on a single core!

Meanwhile humans seem not to have been forging ahead as anticipated. Three years ago I expected new players to emerge who would push the envelope of our strategic understanding of Arimaa. Tuks did manage to beat me out for third place in the 2011 World Championship, but the WHR ratings still show chessandgo, Adanac, and me ahead of Hippo, Nombril, Tuks, and rabbits. If chessando is still only around 2600 strength, humanity has a scant 400-point advantage over the bots. (One could argue for chessandgo being over 2700 based on WHR, but his gameroom rating is below 2500, so the evidence balances out.)

What does everyone else think? Will we hold out past 2020? Am I still underestimating the computers? Please share.

Title: Re: State of the Challenge 2012
Post by Adanac on Jan 17^th, 2012, 9:37am

on 01/16/12 at 20:04:11, Fritzlein wrote:

Yes, it’s becoming apparent that bots will overtake humans much sooner than many people, myself included, originally anticipated. Somewhere between 2018-2024 is my current guess. Humans are improving steadily, and I’m impressed by how much deeper our level of understanding is now versus, for example, back in 2008 – especially considering how small the Arimaa community is. But bots are obviously improving far faster than humans are. Specifically, I’m impressed at how well Marwin & Sharp judge when to use/not to use a camel in an attack or counter-attack, the key moment to advance a rabbit to create goal pressure, when to swarm, and anticipating when/where to preemptively blockade a key defensive square. These are all areas of the game where I feel that the top bots are much stronger than they had been 2-3 years ago. I haven't played as many games against Briareus or Clueless recently but they're getting tougher every year too. I’m still amazed at how easy it is to swindle bots with obvious goal attacks, though. Hopefully the machines will still have a few glaring weakness like that at the end of this decade :)

It’s great that we’ve seen the emergence of a new crop of high-level human players in the past few years, but we’ll need a much deeper pool of Arimaa players in the coming years. I expected we’d have more 2600+ rated players by now and if we don’t get any new prodigies soon there will eventually be more and more pressure on Fritz & Jean every year during the Arimaa Challenge (assuming we’re fortunate enough that they can still devote their time to defending the Challenge in the future).

Title: Re: State of the Challenge 2012
Post by Hippo on Jan 17^th, 2012, 10:00am

Yes I expect the challenge to be beaten till 2020 and I am going to join machine side ;). I hope the challenge will last at least to 2018 to have motivation to make progress of my bot as well. I still hope alpha-beta is not good enough for Arimaa and changes in search methods are required.

Title: Re: State of the Challenge 2012
Post by mistre on Jan 17^th, 2012, 10:11am

on 01/17/12 at 09:37:09, Adanac wrote:

I expected we’d have more 2600+ rated players by now and if we don’t get any new prodigies soon there will eventually be more and more pressure on Fritz & Jean every year during the Arimaa Challenge (assuming we’re fortunate enough that they can still devote their time to defending the Challenge in the future).

It might be too early to rule Boo out as a prodigy. He has only been playing for 7 months and a little over 200 games and is already ranked 11th in WHR for active players...

Title: Re: State of the Challenge 2012
Post by Tuks on Jan 17^th, 2012, 2:23pm

and annoyingly, beats me all the time, haha, i expect he will lead us to break the top three this year, watch out!

Title: Re: State of the Challenge 2012
Post by omar on Jan 17^th, 2012, 11:32pm

If there isn't a breakthrough in the AI software like what's happened in Go with Monte Carlo methods, then I think there is a 50% chance of the challenge being won by 2020 just from incremental improvements in hardware and software.

But the challenge match games will get harder and harder each year to the point where I might have difficulty finding defenders willing to play. Also there is no grantee that the current top players will still be interested in Arimaa until 2020. I think a lot will depend on our ability to attract young players with good potential.

Title: Re: State of the Challenge 2012
Post by Boo on Jan 18^th, 2012, 2:30am

Ahh those nice words ;) Thanks! :)
I am still having fun exploring the game and trying out the new things.

Title: Re: State of the Challenge 2012
Post by megajester on Jan 18^th, 2012, 6:06am

on 01/18/12 at 02:30:55, Boo wrote:

Ahh those nice words ;) Thanks! :)
I am still having fun exploring the game and trying out the new things.

Ahh, look at him, so innocent and cuddly. Just you wait till he gets bigger... GWWRR0O0O0AO0A0OAO0ARRR! RUN MEN, RUN FOR YOUR LIVES!!

Title: Re: State of the Challenge 2012
Post by chessandgo on Jan 18^th, 2012, 8:10am

http://4.bp.blogspot.com/-hr8sICEoYfg/Ts0Fl7jAJlI/AAAAAAAAAI0/z6pjDJR5X5o/s1600/running-away.gif

PS: SPQA, nice!!

Title: Re: State of the Challenge 2012
Post by megajester on Jan 19^th, 2012, 8:08am

on 01/18/12 at 08:10:54, chessandgo wrote:

http://4.bp.blogspot.com/-hr8sICEoYfg/Ts0Fl7jAJlI/AAAAAAAAAI0/z6pjDJR5X5o/s1600/running-away.gif

PS: SPQA, nice!!

Ave Chessandgoix the Gaul, morituri te salutant! (Aut non...)

Title: Re: State of the Challenge 2012
Post by chessandgo on Jan 19^th, 2012, 4:37pm

Hehe!

I do hope you won't die, indeed (hoping I understood your sentence correctly, that is :))

Title: Re: State of the Challenge 2012
Post by Fritzlein on Feb 1^st, 2012, 8:07am

The new top bots whipped the old top bots in their qualifying games. Look at the current gameroom rating list!

2443 chessandgo
2410 Fritzlein
2392 Adanac
2389 bot_marwin
2380 bot_sharp
2303 99of9
2291 bot_briareus

I remember when I first learned about Arimaa in the summer of 2004. I read on Wikipedia and on the arimaa.com home page how this new game was supposedly difficult for computers to play. Then I checked the rating list of the game room, where only three players (99of9, omar, and Belbo) were rated higher than the top bot (bot_bomb). I immediately put a big discount on the claims of computer-resistance, and wondered whether a computer would win the Arimaa Challenge in just a couple of years.

Almost eight years later I am experiencing creepy deja vu. Back in 2004, I eventually learned that the situation was much more subtle than my first impression, because humans had resources that had not yet been tapped. We are much more accustomed to computer progress than human progress, so the latter sometimes takes us by surprise. Also four years of stagnation of the top bot proved that progress on that side was not inevitable.

Now that we have had a few years of amazing bot progress and relative (although certainly not complete) stagnation at the top of the human list, I am forced to wonder what untapped resources we humans have left. We simply must start learning additional strategies to defend the Arimaa Challenge, or be overrun while we stand still. Where will this inspiration come from? From new players? From fencing with the new, stronger bots?

I don't know what is going to happen next, but I would definitely forgive newcomers for having the same first impression that I had, namely that claims of Arimaa's computer resistance are overblown, and it will be just a couple of years before silicon rules.

Title: Re: State of the Challenge 2012
Post by rbarreira on Feb 1^st, 2012, 10:14am

on 02/01/12 at 08:07:10, Fritzlein wrote:

Look at the current gameroom rating list!

2443 chessandgo
2410 Fritzlein
2392 Adanac
2389 bot_marwin
2380 bot_sharp
2303 99of9
2291 bot_briareus

I definitely see your points Fritzlein (nice thread btw), however it should be pointed out that these bots have been playing a lot in Blitz and Fast time controls, which probably over-rates them somewhere between 100-200 points.

This indicates that at the 2 minutes time control, the top bot this year may perform between 2200 and 2300 (my guess would be closer to 2300).

Title: Re: State of the Challenge 2012
Post by Boo on Feb 1^st, 2012, 10:29am

Quote:

This indicates that at the 2 minutes time control, the top bot this year may perform between 2200 and 2300 (my guess would be closer to 2300).

Yeah, it would be interesting to compare human/bot ratings including only 2 min games.
2min games is such a different story compared to 30s.

Title: Re: State of the Challenge 2012
Post by Fritzlein on Feb 1^st, 2012, 10:33am

on 02/01/12 at 10:14:57, rbarreira wrote:

This indicates that at the 2 minutes time control, the top bot this year may perform between 2200 and 2300 (my guess would be closer to 2300).

Yes, the bot ratings at two minutes per move will be lower, and I stand by my estimate that it will be close to 2200 for the top bot even though I was shocked by the latest rating list. The influence of the time control is one of the things I didn't immediately grasp when I saw the rating list in 2004 either. (The top bot for a while was actually not bot_bomb but bot_speedy, which was bomb playing at a faster time control.) In addition to the time control, I expect that the game room ratings of the top humans are a bit low at present compared to the general scale of the game room ratings. I expect that if the seven players at the top of the list played a twenty-fold round-robin even at blitz, the humans would gain net rating points.

That said, deciding whether the human advantage under Challenge conditions is presently 200 points or 400 points is less important in the long run than the relative rate of change. Even if I am right that the current gap is 400 points, I believe bots will make up that gap in eight years if humans stand still. If we are to defend the Arimaa Challenge, we simply must raise our game to a new level.

I personally believe that I am capable of playing at a substantially higher level than I do now, but achieving that goal stands in conflict with passing my classes. :D

Title: Re: State of the Challenge 2012
Post by mistre on Feb 1^st, 2012, 3:03pm

on 02/01/12 at 10:33:45, Fritzlein wrote:

I personally believe that I am capable of playing at a substantially higher level than I do now, but achieving that goal stands in conflict with passing my classes. :D

Ahh, that is where the bots have the advantage. They don't have time constraints on their play and they don't get rusty when they don't play.

I think a big factor on what year the challenge actually falls is how much the top players are able to play.

Title: Re: State of the Challenge 2012
Post by tize on Feb 2^nd, 2012, 3:34pm

Here is my predictions/thoughts about the challenge:
* A bot will eventually win the challenge
* It won't be until 2015 that a top 10 player from the WHR list loses.
* All years after the first top 10 player falls then the challenge bot will always win at least one of the best of three matches.
* The bot winning the challenge will be alpha-beta based.

Title: Re: State of the Challenge 2012
Post by Fritzlein on Feb 3^rd, 2012, 5:58am

on 02/01/12 at 10:33:45, Fritzlein wrote:

I expect that if the seven players at the top of the list played a twenty-fold round-robin even at blitz, the humans would gain net rating points.

Ho ho, so I say, and then last night manage only a 2-2 split with Sharp2011Blitz, last year's version running on a single core. I need to stop underestimating the bots.

Title: Re: State of the Challenge 2012
Post by Fritzlein on Feb 4^th, 2012, 11:56am

on 01/16/12 at 20:04:11, Fritzlein wrote:

Meanwhile humans seem not to have been forging ahead as anticipated. Three years ago I expected new players to emerge who would push the envelope of our strategic understanding of Arimaa. Tuks did manage to beat me out for third place in the 2011 World Championship, but the WHR ratings still show chessandgo, Adanac, and me ahead of Hippo, Nombril, Tuks, and rabbits.

Shame on me for having left hanzack out of this calculation. Now that he has knocked off chessandgo and Adanac in successive rounds, it is starting to look like some new player is indeed pushing the envelope of human understanding. Also I have belatedly noticed that hanzack's 1-7 record against Sharp2001Blitz is really a 6-2 record, with wins in a wide variety of position types, and with crushing wins on the tail end. He is dominating the bot against which I can barely eke out a 50% score. (I was 3-3 again last night.) I'm certainly going to do my best to beat hanzack if we meet in later rounds of the World Championship, but my personal interest in winning aside, it would be fantastic for Arimaa if we were to have a new human champion. Hurray for progress!

Title: Re: State of the Challenge 2012
Post by mistre on Feb 4^th, 2012, 2:10pm

on 02/04/12 at 11:56:16, Fritzlein wrote:

Also I have belatedly noticed that hanzack's 1-7 record against Sharp2001Blitz is really a 6-2 record, with wins in a wide variety of position types, and with crushing wins on the tail end.

This was the point I was trying to make about him resigning games. At first it disguised his rating, but now that he has to play all games unrated except event games, his rating will eventually go up. However, if he keeps resigning games, it makes it very difficult to figure out his actual strength vs various bots.

Title: Re: State of the Challenge 2012
Post by aaaa on Feb 8^th, 2012, 4:45pm

My prediction is that if bots ever were to become stronger than humans, that the transition will not be clean; here, this would mean that a bot would win the Challenge in such a way, that it will come across as a bit of a fluke and a slight embarrassment, it being pounded back into submission by vengeful members of Homo Sapiens afterwards as its weaknesses become manifest.

Title: Re: State of the Challenge 2012
Post by rbarreira on Feb 8^th, 2012, 5:52pm

on 02/08/12 at 16:45:09, aaaa wrote:

Not a bad prediction. But I don't think it would be that embarassing if a non-learning challenger winner had exploitable weaknesses, such is the nature of static bots.

The more embarassing scenario I can think of is the challenge being won only/mostly due to things such as timeouts (that would be bad...) or terrible blunders a la Kramnik...

Title: Re: State of the Challenge 2012
Post by Fritzlein on Feb 8^th, 2012, 11:13pm

on 01/17/12 at 10:11:14, mistre wrote:

It might be too early to rule Boo out as a prodigy. He has only been playing for 7 months and a little over 200 games and is already ranked 11th in WHR for active players...

And in the three weeks since you wrote that, Boo moved up to 9th, hanzack beat chessandgo and Adanac back-to-back to jump up to 3rd, and tharkun appeared from nowhere to beat the bot ladder. Maybe the next wave of human improvement is happening right now just to teach me not to complain that it isn't happening!

Title: Re: State of the Challenge 2012
Post by tharkun on Feb 9^th, 2012, 2:47pm

on 02/08/12 at 23:13:27, Fritzlein wrote:

Maybe the next wave of human improvement is happening right now just to teach me not to complain that it isn't happening!

Who knows... I think the best chance to push the limits of human strategic understanding and general playing strength is an increase in the total number of Arimaa players. A pyramid can reach higher when the base is larger. Or maybe we could try to convert players of go, chess and the like. I think it is no coincidence that Boo has risen fast through the ranks. I noticed a chess grandmaster among our new members and he is still on 100% so far; if he starts to play Arimaa for earnest he might go places ;D

Title: Re: State of the Challenge 2012
Post by Boo on Feb 19^th, 2012, 9:12am

Quote:

Be the first to get one of your rabbits to the other side of the board.

I start thinking that this rule is favouring computers a lot. It creates lots of tactical possibilities and contradicts to the game idea of 'easy for human, hard for computer'.

If it was changed to something like 'Friendly rabbit on the other side of the board is lost if no other friendly piece is sitting next to it.', that would create much more strategical play. So that in order to score a goal, a player would need one of his pieces and a rabbit next to it.

The endgame should remain strategical.

Title: Re: State of the Challenge 2012
Post by Nazgand on Feb 19^th, 2012, 11:19am

I have a proposition for 'the hanzack problem'.
We could start with the assumption that if there is a forced win in 1, hanzack noticed it.
So we could have a script that would mark all his games as won if there is at any point in the game a forced win in one. This would, no doubt, cause him to start winning by forced win in two without having previously done a forced win in one.

When he becomes an expert at that, we write a script to mark all his forced win in two as won, so that he will start winning by forced win in three.

We repeat this until he is consistently winning by forced win at setup, where he would have completed his training and would be the most powerful Arimaa player ever.

Title: Re: State of the Challenge 2012
Post by chessandgo on Feb 19^th, 2012, 11:24am

on 02/09/12 at 14:47:15, tharkun wrote:

I noticed a chess grandmaster among our new members and he is still on 100% so far; if he starts to play Arimaa for earnest he might go places ;D

Really, who's this?

Title: Re: State of the Challenge 2012
Post by chessandgo on Feb 19^th, 2012, 11:26am

At some point he'll be resigning games on move 1 as Silver, and we'll say: hey, Arimaa is a win for Silver!

Title: Re: State of the Challenge 2012
Post by tharkun on Feb 19^th, 2012, 5:00pm

on 02/19/12 at 11:24:33, chessandgo wrote:

Really, who's this?

His nickname is Carrot. He was introduced to Arimaa by Harren.

Title: Re: State of the Challenge 2012
Post by Fritzlein on Feb 19^th, 2012, 8:12pm

on 02/19/12 at 09:12:58, Boo wrote:

I start thinking that this rule is favouring computers a lot. It creates lots of tactical possibilities and contradicts to the game idea of 'easy for human, hard for computer'.

You are right that the endgame is so tactical that it favors the computers. For humans to beat the best bots, we have to win the opening and middlegame.

I am afraid, though, that your suggestions to change the rules are too late. For better or worse, we are stuck with the Arimaa rules as they currently exist. You must know this from chess: even though chess is broken, with too many draws among grandmasters, try suggesting that the stalemate rule should be changed, and you will get nowhere. :o

I am very pleased that Arimaa is not showing any flaws, except perhaps not being as computer-resistant as Omar hoped. We don't have color imbalance, we don't have stereotyped openings, we don't have draws or stalemates, while we do have games that are tense and dramatic, featuring a variety of styles and conflicting strategic priorities that are still being worked out. If the only thing wrong with Arimaa is that computers eventually win the Arimaa Challenge, I'd say we are doing pretty well. :)

Title: Re: State of the Challenge 2012
Post by Boo on Feb 20^th, 2012, 1:53am

Quote:

I am afraid, though, that your suggestions to change the rules are too late.

Why do you think it is too late? The arimaa is a baby-game compared to chess which is thousand years old. :)

Quote:

Arimaa demonstrates that when it comes to serious game playing, humans still dominate. By offering this challenge we hope to increase the awareness that the difficult problem of teaching computers to play strategy games has not yet been tackled. There is much work that needs to be done in understanding how humans play strategy games and producing similar capability in software. We hope this challenge will spur some new and radically different ways of replicating this astonishing human capability in software. The breakthroughs that will come from such research can have significant applications in a wide variety of fields.

Now look what is happening. Bot developers are not teaching AI strategy, but teaching bots to be strong in tactics! They are adopting the same algorithms that are known to work in chess. How is it going to spur replicating human capabilities? No way... The challenge is flawed.

FLAWED! FLAWED! FLAWED!

Title: Re: State of the Challenge 2012
Post by chessandgo on Feb 20^th, 2012, 5:11am

on 02/19/12 at 17:00:05, tharkun wrote:

His nickname is Carrot. He was introduced to Arimaa by Harren.

Ok, thanks! He's very young also, that's nice.

Title: Re: State of the Challenge 2012
Post by Fritzlein on Feb 20^th, 2012, 10:53am

on 02/20/12 at 01:53:40, Boo wrote:

Sure, I don't mind if you think the Arimaa Challenge is flawed. My point was only that Arimaa per se is flawless. :)

Title: Re: State of the Challenge 2012
Post by tize on Feb 20^th, 2012, 2:12pm

on 02/20/12 at 01:53:40, Boo wrote:

I understand what you mean, and I am guilty of adopting known algorithms. :-[

But I think there is a lot of room for free and inovative thinking that can be done inside the alphabeta algorithm. Like training evaluation functions, pruners, tactical threat analysers etc. This will probably never replicate human thinking but it might still advance the field of AI.

So I don't think that the challenge is flawed, but instead a possible way of finding one missing piece of the big AI puzzle.

Title: Re: State of the Challenge 2012
Post by aaaa on Feb 20^th, 2012, 2:56pm

I don't think any stigma should be attached to the development of bots in which only conventional techniques feature prominently. At the very least, they actually put the claim of Arimaa's supposed computer intractability to the test and they additionally serve as excellent benchmarks to gauge the effectiveness of non-orthodox approaches when these do happen to be tried out (which was the original motivation behind OpFor).

Title: Re: State of the Challenge 2012
Post by rbarreira on Feb 20^th, 2012, 3:11pm

There is still progress to be made using existing techniques. When progress stops with these techniques it will be time to find ways to achieve what these techniques can't.