Author |
Topic: State of the Challenge 2009 (Read 5006 times) |
|
Fritzlein
Forum Guru
Arimaa player #706
Gender: 
Posts: 5928
|
 |
State of the Challenge 2009
« on: Mar 2nd, 2009, 11:49am » |
 Quote  Modify
|
Omar's comments in the 2009 Computer Championship thread seem like a good excuse to kick off the 2009 Arimaa Challenge thread with some speculation about the current man vs. machine balance of power. I propose that from now on we have a yearly "State of the Challenge" discussion. It will be a fun tradition to formalize what has been happening informally over the years. For example, I wish I could find 99of9's comments from 2004 about how he rated the chances of Omar paying out the $10,000 before 2020 to be significantly more than 50%. Those odds seem to have dropped yearly with Bomb ruling the roost, but have they bounced back significantly this year? Here's what Omar said:
on Mar 1st, 2009, 9:08pm, omar wrote:| Humans have enjoyed a six year plateau in bot performance and perhaps gotten a little over confidence in our ability to stay ahead of the bots. Now the fun is really going to begin and we are going to be tested like never before. My expectation is that humans are going to learn a lot about tactics from the bots. Also we are going to learn something intrinsic about the Arimaa game tree. First we are going to notice that there are areas of the game tree that are pro tactics and areas that are pro strategy. Then the real battle for humans is going to be to see if they can steer the game into a pro strategy area while the bots try to steer it into a pro tactics area. If the Arimaa game tree is intrinsically more pro strategy then the humans will be able to maintain the lead; otherwise the bots will close the gap and potentially surpass us just by riding the hardware improvement ramp. |
|
I mostly agree with Omar's analysis. We have gotten overconfident from using Bomb as a nearly-stationary benchmark while we humans gradually improved. I'd call it four years though, not six; Bomb was definitely still improving in early 2005. After that Bomb's slight improvement came only from running on a faster chip, and even that improvement was minimized because Bomb wasn't parallelized.
OpFor's incredible showing in the 2008 Postal Mixer was our first wakeup call. In hindsight I think we can rule out the hypothesis that OpFor's wins were due to simple overconfidence on our part. Ron Weasley had already sounded the alarm early in the tournament when several of OpFor's positions were still unclear, and which humans should still have won if paying attention was the only requirement. But there was something deeper going on; OpFor actually played quite well in "wild" positions with multiple traps contested.
I definitely think that we will have to improve our strategic jugdment in multi-trap positions, and hone our tactics at trap control fights. Nevertheless, for the present I don't believe that humans will have to avoid certain types of positions in order to stay ahead of computers. It would be a severe disadvantage for us to have to play tame positions only, because then the struggle would mostly be about making the position wild or preventing it from becoming so. But the fact that a position is wild does not yet mean that a bot will win from either side. For example, a capture race that turns into a goal race definitely fits into the loose definition of tactical that Omar presents, but winning such a race against a bot is entirely possible, as blue22 has demonstrated repeatedly. If blue22 can chop to goal in about four turns in the northeast while a bot can chop to goal in about six turns in the southwest, it doesn't matter if the bot is a little better than him at racing; blue22 is still going to win the race.
My point is that strategic judgment stands at the beginning of every tactical race. I don't think the bots can beat us with a mantra of "always complicate", because very often the move that makes the position wilder also concedes some disadvantage. For example, I suspect that part of the new strength of clueless is that it is eager to enter races rather than remaining tame (like Bomb) when the elephants are deadlocked. But clueless totally lacks the ability to distinguish a favorable race from an unfavorable one.
Take my recent win over clueless in game 97957. I deadlocked our elephants early with a smother, but the smother isn't worth much since it takes my elephant to hold in clueless' elephant. There's hardly any hope for me to rotate my elephant out of the smother, particularly since clueless is now smart enough not to advance rabbits into that situation. So what did clueless do? It activated its camel. As a general principle, this is smart for clueless to do. When elephants are deadlocked, you should get more free with your camel. But in this particular case the smart general principle was executed stupidly. The camel hostage that clueless offered to me was worth significantly more than the elephant blockade I had to give up to freeze the camel.
In other words, clueless did exactly what Omar is suggesting bots need to do, by turning a strategic situation into a tactical one. Clueless didn't sit back and let me exploit my elephant smother, it quickly complicated the game. But the ensuing capture race is one I am going to win, even though clueless is better than me tactically. I'm going to win that race because my elephant is next to his camel, while his elephant is only next to my horses. The fact that I won a camel for a horse has little to do with great tactics on my part during the capture race; it was mostly about bad strategic judgment from clueless to offer me such a favorable race, and good strategic judgment from me to immediately switch gears and ditch the smother for something better.
The 2009 versions of clueless, OpFor, and GnoBot have definitely raised the bar, and further such improvements in the years to come will make the Arimaa Challenge interesting again. We humans have to keep learning to stay ahead. I do not, however, believe we are at the point that we can beat bots only in quiet positions. Our learning does not have to be focused on how to prevent complications from ever occurring. On the contrary, I predict we still have years of being able to punish bots that complicate at all costs, by learning how to snap off their attacks and make them pay for being overly aggressive.
|
| « Last Edit: Jul 3rd, 2009, 12:17pm by Fritzlein » |
 IP Logged |
|
|
|
omar
Forum Guru
Arimaa player #2
Gender:
Posts: 1003
|
|
Re: 2009 Arimaa Challenge
« Reply #1 on: Mar 3rd, 2009, 1:04pm » |
Quote Modify
|
I really like the idea of having a yearly "State of the Challenge" discussion. Maybe we should rename the thread similarly so that the current name can be used for discussion of the actual challenge match games, announcement of defenders, etc. Also makes searching for these discussions easier.
I am certainly not a good tactical player and perhaps I am using my own weakness as a basis for predicting that human players will want to steer the game into pro strategy positions. Players like Fritzlein and chessandgo have an amazing ability to handle tactical positions. Perhaps as the bots become stronger such players will continue to learn from the bots and stay ahead. However, players like me, who hardly calculate more than a ply or two on just a few selected moves will get left behind as the bots continue to get better tactically. Fo me being able to steer the game into pro strategy positions will become essential as the bots get better.
|
|
IP Logged |
|
|
|
Fritzlein
Forum Guru
Arimaa player #706
Gender:
Posts: 5928
|
|
Re: 2009 Arimaa Challenge
« Reply #2 on: Mar 3rd, 2009, 4:53pm » |
Quote Modify
|
on Mar 3rd, 2009, 1:04pm, omar wrote:| Maybe we should rename the thread similarly |
|
Let's wait to rename the thread until at least one other person has chimed in with an opinion on the State of the Challenge. 
Quote:| However, players like me, who hardly calculate more than a ply or two on just a few selected moves will get left behind as the bots continue to get better tactically. Fo me being able to steer the game into pro strategy positions will become essential as the bots get better. |
|
What if CeeJay had said in 2005 that the only way to stay ahead of the bots is to find a trick like bait-and-tackle? Sure, bait-and-tackle might have been the only way he could reliably beat Bomb, but that didn't mean it was the only avenue for humanity to explore. And sure enough, lots of players who hadn't even discovered Arimaa at that point are now able to reliably beat Bomb without the bait-and-tackle.
I think your pro-strategy/pro-tactics distinction is not only short-changing the subtlety of Arimaa, it is also short-changing your ability to learn strategies from other players. You shouldn't suppose that the next method for staying ahead of the bots someone else comes up with will be beyond your ability to execute. I suspect that over the next few years you are more likely to broaden your arsenal of strategies you use against bots than you are to shrink your arsenal into trying to force the game into a small handful of position types.
|
|
IP Logged |
|
|
|
Janzert
Forum Guru
Arimaa player #247
Gender:
Posts: 1016
|
|
Re: 2009 Arimaa Challenge
« Reply #3 on: Mar 3rd, 2009, 8:24pm » |
Quote Modify
|
Last year at this time I thought computers had about a 10% chance of winning. But with the performance of OpFor in the postal mixer and then the great leaps Clueless and Gnobot have made I'm hovering up around 50% now.
Janzert
|
|
IP Logged |
|
|
|
99of9
Forum Guru
Gnobby's creator (player #314)
 
Gender:
Posts: 1413
|
|
Re: 2009 Arimaa Challenge
« Reply #4 on: Mar 3rd, 2009, 9:16pm » |
Quote Modify
|
on Mar 2nd, 2009, 11:49am, Fritzlein wrote:| I wish I could find 99of9's comments from 2004 about how he rated the chances of Omar paying out the $10,000 before 2020 to be significantly more than 50%. |
|
Maybe you're referring to the forum post linked below. Your memory has been kind to my predictions! Re-reading this thread makes entertaining reading, there are now-humorous prophecies from a number of people.
http://arimaa.com/arimaa/forum/cgi/YaBB.cgi?board=talk;action=display;nu m=1128966698;start=
Nevertheless, I'm still an optimist, and certainly consider the 2020 chances nearer 50% than 10%.
|
| « Last Edit: Mar 3rd, 2009, 9:18pm by 99of9 » |
IP Logged |
|
|
|
Fritzlein
Forum Guru
Arimaa player #706
Gender:
Posts: 5928
|
|
Re: 2009 Arimaa Challenge
« Reply #5 on: Mar 4th, 2009, 6:38am » |
Quote Modify
|
Thanks for the link, 99of9. It is indeed amusing to go back and read our old predictions from time to time.
I wanted to expand on something that occurred to me in chat with aaaa yesterday. Suppose that we had a good way to measure true skill at Arimaa, something like the game room ratings but more accurate. Suppose that we plotted the gap in true skill between the top human and the top bot as a function of time. What would we expect the graph to look like?
The image floating in my mind is something like the graph of f(x) = log(x+1) - x/10. The gap started near zero, and has risen dramatically, but at some point our advantage will level out, and after that it will get smaller every year, almost linearly, until we lose to computers. After that computers will get so much better than us it won't even be funny.
I wonder if the same sort of image isn't influencing other people's predictions as well. If you believe that there will be one turning point in the graph, then predicting the Challenge is a matter of reading when the turning point has come, and guessing how sharply the U-turn will happen. Omar's comments seem to be as good as saying that the tide has already turned, and from here on out humans are on the downward slope.
I believe in the one-turning-point model as a rough approximation. However, it is not easy to predict the turning point within that model because the actual graph is not smooth when you look at it close up. Both human learning and bot improvement happen in a series of small leaps rather than in a smooth ascent. Plotted over a long enough period the increase in human playing strength and the increase in bot playing strength will look like smooth curves, but in the short run the graphs will be stair-stepped.
Looking at bot strength plotted over the past four years we have at first a nearly flat graph of Bomb2005 being the benchmark and getting better only from running on a faster chip, followed by a spike in the last few months as clueless (and OpFor and GnoBot?) surged past Bomb. I admit that we probably got overconfident due to the flat part of the bot-strength graph, but by the same token it would be unreasonable panic to take the recent rate of improvement as the new normal. If you look how much bots have improved in the past four years rather than in the past four months, the graph is going to have a very different slope, and your long-term predictions will come out very differently.
Because of the stair-step nature of bot improvements, I'm still claiming we haven't reached the turning point in the man-machine gap. Of course my prediction also assumes that human playing strength has not yet plateaued, but I already went into that in my previous posts in this thread. The human strength curve is probably even harder to predict accurately than the bot strength curve, so we'll just have to wait and see. In another four years 99of9 can link back to this post and we can have another good laugh. 
|
|
IP Logged |
|
|
|
omar
Forum Guru
Arimaa player #2
Gender:
Posts: 1003
|
|
Re: 2009 Arimaa Challenge
« Reply #6 on: Mar 4th, 2009, 6:49am » |
Quote Modify
|
on Mar 3rd, 2009, 4:53pm, Fritzlein wrote:
You shouldn't suppose that the next method for staying ahead of the bots someone else comes up with will be beyond your ability to execute. I suspect that over the next few years you are more likely to broaden your arsenal of strategies you use against bots than you are to shrink your arsenal into trying to force the game into a small handful of position types. |
|
Not at all. I can definitely learn new methods and strategies and in fact will need to because the bots become better at avoiding the ones that used to work. For example its harder to get clueless to fall for bait-and-tackle.
Also I don't mean to classify all positions as either pro-tactic or pro-strategy. It's really a spectrum; I was just referring to the extremes.
|
|
IP Logged |
|
|
|
Fritzlein
Forum Guru
Arimaa player #706
Gender:
Posts: 5928
|
|
Re: 2009 Arimaa Challenge
« Reply #7 on: Mar 4th, 2009, 8:04am » |
Quote Modify
|
But do you think that the tide has turned and the human advantage will get smaller from this year forward?
|
| « Last Edit: Mar 4th, 2009, 8:05am by Fritzlein » |
IP Logged |
|
|
|
Fritzlein
Forum Guru
Arimaa player #706
Gender:
Posts: 5928
|
|
Re: 2009 Arimaa Challenge
« Reply #8 on: Mar 4th, 2009, 6:41pm » |
Quote Modify
|
OK, I need to backpedal a little. The game clueless vs GnoBot in Round 3 of the Computer Championship was mind-blowing and scary. I think I might have lost that to either bot from either side. So maybe, yes, there are types of positions I'm going to have to steer clear of.
|
|
IP Logged |
|
|
|
99of9
Forum Guru
Gnobby's creator (player #314)
Gender:
Posts: 1413
|
|
Re: 2009 Arimaa Challenge
« Reply #9 on: Mar 4th, 2009, 7:35pm » |
Quote Modify
|
Some specific estimates and predictions:
40% chance of the challenge falling by 2013
60% chance by 2020
In the year it falls, surprise will be an essential element of the bot's win. That is, after losing, humans will find areas of weakness that will enable them to win reasonably often.
In the year it falls, the strongest human opponent will hang a piece during one of his/her losses.
By 2013, at least one defender will have lost 0-3.
If the challenge has not fallen by the 2018 challenge, developers will then join forces in a final push for the last two years.
|
|
IP Logged |
|
|
|
Fritzlein
Forum Guru
Arimaa player #706
Gender:
Posts: 5928
|
|
Re: 2009 Arimaa Challenge
« Reply #10 on: Mar 5th, 2009, 5:58am » |
Quote Modify
|
Nice predictions, 99of9. It takes guts to get specific; you inspire me to respond in kind.
By mid-2011 (that is to say within two years of Arimaa's commercial release) there will be some new player (that is to say someone who doesn't even know how to play Arimaa at the time I am writing this) who is 200 Elo points better than chessandgo and I are today. S/he will achieve a game room rating of 2700 playing against only humans. Equivalently, in case there is more club play and less on-line play at that time, s/he will be able to win three of four tournament games against chessandgo and me, unless the two of us have similarly improved to keep up.
By 2020 there will be at least one human player with a playing strength of 3000 on the current rating scale.
|
| « Last Edit: Mar 5th, 2009, 5:59am by Fritzlein » |
IP Logged |
|
|
|
Adanac
Forum Guru
Arimaa player #892
Gender:
Posts: 635
|
|
Re: 2009 Arimaa Challenge
« Reply #11 on: Mar 5th, 2009, 11:50am » |
Quote Modify
|
on Mar 5th, 2009, 5:58am, Fritzlein wrote:Nice predictions, 99of9. It takes guts to get specific; you inspire me to respond in kind.
By mid-2011 (that is to say within two years of Arimaa's commercial release) there will be some new player (that is to say someone who doesn't even know how to play Arimaa at the time I am writing this) who is 200 Elo points better than chessandgo and I are today. S/he will achieve a game room rating of 2700 playing against only humans. Equivalently, in case there is more club play and less on-line play at that time, s/he will be able to win three of four tournament games against chessandgo and me, unless the two of us have similarly improved to keep up.
By 2020 there will be at least one human player with a playing strength of 3000 on the current rating scale. |
|
My predictions are similar to Fritzlein's. We have a tiny, tiny Arimaa player pool, with only 18 people on the entire planet willing to participate in the 2009 World Championship. Once the game gets marketed and our player pools grows exponentially we’ll have dozens of players with 2600+ ratings. And perhaps a few of those will reach the 3000 level. I’ll predict that 3 players will have 3000+ ability by 2015 (measured by today’s standards).
The bot developers have a huge advantage at this moment in time because unlike chess or go where the ratio of active players to developers is perhaps 100000:1 or more, in Arimaa the ratio might be 10:1 (?). Once we recruit a deeper pool of human players, the gap between humans and bots will grow much larger.
I’ll predict that the best Arimaa bot will become permanently better than the best human in 2028. I will estimate there’s a 10% chance that a bot developer will win the Arimaa Challenge within the next 5 years. Even if a bot developer pulls it off, so many super-geniuses will become hooked on Arimaa in the mid-2010s that humans will once again reign supreme and many years will elapse before bots close the gap again.
|
|
IP Logged |
|
|
|
Fritzlein
Forum Guru
Arimaa player #706
Gender:
Posts: 5928
|
|
Re: 2009 Arimaa Challenge
« Reply #12 on: Mar 5th, 2009, 12:51pm » |
Quote Modify
|
on Mar 4th, 2009, 7:35pm, 99of9 wrote:40% chance of the challenge falling by 2013
60% chance by 2020 |
|
These prediction percentages translate into a declining per-year chance of the Challenge falling. By Bayes' Law, you are implying that if the Challenge hasn't fallen in 2009-2013, there is a 33% chance that it will fall in 2014-2020.
Assuming that the per-year chance of the Challenge falling is constant in the first epoch and again constant in the second epoch, then you are saying there is a 9.7% chance for bots to win each year from 2009-2013, and a 5.6% chance for bots to win each year 2014-2020.
Do you really feel that the per-year chance of the Challenge falling will decline over time? My hunch is that the per-year odds for the bots will decline for a few more years, but then get start to get better again each year as we approach 2020, because human learning will plateau at some point, although it hasn't yet.
Adanac's scenario that bots might win in the short run and then be left in the dust by humanity the following decade is yet a third option that seems far from either my intuition or 99of9's. This thread is turning into a good State of the Challenge thread after all. I have re-titled my first post to give the thread a new name.
|
| « Last Edit: Mar 5th, 2009, 12:55pm by Fritzlein » |
IP Logged |
|
|
|
mistre
Forum Guru
Gender:
Posts: 553
|
|
Re: State of the Challenge 2009
« Reply #13 on: Mar 5th, 2009, 6:06pm » |
Quote Modify
|
Doesn't the percent chance of the challenge being met depend on the human opposition? Obviously if we have our three current and most active highest rated players (Fritzlein, Chessandgo, and Adanac) defending then we have a much better chance then if someone like me or a lower rated player is called to defend (I played one match last year).
If we can keep the top 3 players active and interested in always defending (considering also that there may be some new players out there that could get close to a 3000 rating, at least 2700 does not seem out of the question once the user base expands) then I think the chances are pretty good that we can make it to 2020 without losing the challenge.
On the other hand, if Omar insists on spacing out the challengers and letting some lower rated players assist in defending, then I swing the favor back to the bots. I have yet to play the latest versions of Clueless or Gnobot and wonder if I would even be 50-50 with them.
|
|
IP Logged |
|
|
|
Fritzlein
Forum Guru
Arimaa player #706
Gender:
Posts: 5928
|
|
Re: State of the Challenge 2009
« Reply #14 on: Mar 5th, 2009, 6:47pm » |
Quote Modify
|
on Mar 5th, 2009, 6:06pm, mistre wrote:| On the other hand, if Omar insists on spacing out the challengers and letting some lower rated players assist in defending, then I swing the favor back to the bots. |
|
Spreading out the Challenge defense to lots of players is not a requirement; it is a gesture of confidence similar to offering a piece handicap when you expect to win anyway. My hunch is that if the Challenge starts to seem in jeopardy, not only will the handicaps disappear, the names of the defenders will get a lot more predictable.
|
|
IP Logged |
|
|
|
|
Home
Help
Search
Members
Login
Register
Reply
Notify of replies
Send Topic
Print