Arimaa Forum - Print Page


    
      
        Arimaa Forum
        (http://arimaa.com/arimaa/forum/cgi/YaBB.cgi)
      

        Arimaa >> Events >> 2014 State of the Challenge
        
(Message started by: Fritzlein on Feb 20^th, 2014, 2:39pm)

Title: 2014 State of the Challenge
Post by Fritzlein on Feb 20^th, 2014, 2:39pm

We haven't had this discussion since the 2012 State of the Challenge thread (http://arimaa.com/arimaa/forum/cgi/YaBB.cgi?board=events;action=display;num=1326765851), but it looks like it is time for everyone to weigh in once again, as evidenced by Ail's post in another thread:

on 02/19/14 at 18:08:57, Ail wrote:

syed 2529
bot_sharp 2524
Fritzlein 2495
chessandgo 2433
Adanac 2360
browni3141 2354

Sharp is in 2nd place with only 5 ELO below the first human.
That does certainly not look like a comfortable advantage.

So is mankind in danger to lose Arimaa to the machines as well?

I remember having much the same feeling when I joined Arimaa in 2004, except that back then it was a version of Bomb that was not far from the top of the rating list. That didn't look like human dominance to me! I didn't realize how volatile the ratings were, and how much impact the time control had. Sharp's current rating is mostly from 30s/move and 15s/move games.

When I looked closer, I saw that Omar had won the 2004 Arimaa challenge eight games to zero, without being in danger in any of them. That was more like the dominance I had heard touted.

But the biggest thing I didn't know in 2004 (because nobody else knew it either) was exactly how much better human players would get over the next few years. The real kicker was not the balance of power at that exact moment, but the rate at which the two sides improved from there. Human skill increased by leaps and bounds, whereas computer skill (at least at first) increased more slowly.

on 02/19/14 at 19:14:51, browni3141 wrote:

In reality, sharp is probably several hundred points behind the top player at a 2min/move time control.

Indeed, the gameroom rating of the top computer during the 2013 screening was just 2121, and sharp didn't even qualify for the screening! That would suggest that top human vs. top bot favors the human by about 400 Elo.

But the trend remains important. Between 2012 and 2013, there was apparently not much improvement in any of the top bots. This year lightvector has already said he put in more time on improvements than last year. Possibly sharp's astronomical rating reflects an actual strength increase?

Everyone please feel free to chime in with predictions, so that you can look back in a few years and be embarrassed. I have stuck my neck out a few times already, but at the moment I will stand pat from two years ago: "I now put the chance of Omar having to pay out his $10,000 prize at 30%."

There has been some progress on both sides in the last two years, but I feel less on the human side. On the other hand, the clock is starting to run out for the computers. So I still say 30%.

Title: Re: 2014 State of the Challenge
Post by browni3141 on Feb 20^th, 2014, 3:58pm

2020 isn't far away! Although there seems to be some stagnation on both sides, this should favor humans, as we currently maintain a lead on bots. Also, bots need to not only become better than the top humans to have a strong chance, they need to leave no exploitable holes in their strategy which a bot-basher could expose and take advantage of.
I would guess that bots have around a 7.5% chance to win the Challenge, and I don't think it will be any of the current top bots if it happens.
I would love to see one of our defenders pull off another horse handicap this year as a demonstration of human superiority! I think you guys can do it!

@Fritz: if the difference between top humans and top bots has changed little, in your opinion, then wouldn't the chance of them coming out victorious have also decreased, in your opinion?

Edit: Okay, you say you think that humans have improved less since then (which seems contrary to your opinion last year), but I still think the rate of improvement of bots is too small to be worth the lost time.

Title: Re: 2014 State of the Challenge
Post by Fritzlein on Feb 22^nd, 2014, 6:45pm

on 02/20/14 at 15:58:37, browni3141 wrote:

Also, bots need to not only become better than the top humans to have a strong chance, they need to leave no exploitable holes in their strategy which a bot-basher could expose and take advantage of.

Apparently most people give this more weight than I do. My own take is that when the general level of play by top bots catches up to the level of play by top humans, any remaining holes will be easy to plug. My vague impression from chess was that humans put way too much faith in "anti-computer" play. For example, in Kasparov vs. Deep Blue, anti-computer play actually backfired.

Title: Re: 2014 State of the Challenge
Post by Fritzlein on Feb 22^nd, 2014, 7:19pm

on 02/22/14 at 10:21:03, hyperpape wrote:

I ask because the computergo list had an off-topic discussion about whether go or arimaa would fall to bots first.

How about a link? I'm curious what folks are saying. If it moves the discussion along in that thread, my opinion is that currently top computer versus top human is about 10% to win an individual game at 2min/move in their first encounter. On successive encounters, the human winning probability goes up, but to me the first encounter is more indicative of the relative strength.

As said above, the more interesting (and more difficult) question is the rate of improvement on each side. Arimaa's trump card relative to Go is that humans have much room to improve at Arimaa, whereas it will be difficult for a Go player to ever be better than today's best. Arimaa's downfall may be that there is so much room for Arimaa software to improve. :)

Title: Re: 2014 State of the Challenge
Post by Ail on Feb 22^nd, 2014, 7:30pm

First of all let me point at the Top-Rated players again, where Sharp now has actually gotten into first place:

bot_sharp 2559
syed Arifuddin 2529

According to lightvector it beats its predecessor in about 75% of the games which, as someone pointed out, equals an ELO-increase of roughly 190 over that version.

If in chess a program would make a similar-leap, the computer-chess-scene would be very impressed.

Go also has an advantage speaking for it: A broader selection of players to compete for mankind.

I have no idea how the relation between computer-go-developers:go-players and computer-arimaa-developers:arimaa-players is.

Probably not in Arimaa's favour either.

Since I haven't seen any game between a top-human and a top-bot, it's extremely hard for me to make any predictions.

I hope mankind will prevail!

Title: Re: 2014 State of the Challenge
Post by hyperpape on Feb 22^nd, 2014, 9:28pm

Fritz, I see your thoughts match mine. I said currently the best humans had 90% or better chances against the best bots I know about (2012 challenge), which is a much better chance for the bots in Arimaa than in Go. But humans could advance in the next few years, whereas there's much less headroom in Go.

Here is a link, but the discussion is very cursory so far http://dvandva.org/pipermail/computer-go/2014-February/thread.html

For those who know anything about go, bots can now beat good professionals while taking a four stone handicap. That represents enormous progress since 2005; but we may be at a plateau.

Title: Re: 2014 State of the Challenge
Post by Janzert on Feb 22^nd, 2014, 9:45pm

on 02/22/14 at 19:19:21, Fritzlein wrote:

How about a link?

The recent messages are here: http://dvandva.org/pipermail/computer-go/2014-February/thread.html#6525

but that first message is actually reviving a short thread from December: http://dvandva.org/pipermail/computer-go/2013-December/thread.html#6410

Janzert

Title: Re: 2014 State of the Challenge
Post by browni3141 on Feb 22^nd, 2014, 10:39pm

on 02/22/14 at 19:19:21, Fritzlein wrote:

My estimate for a bot's winning chances against me was 5%, so I'll gladly take odds.
Or I'll offer even money that I can beat 10 bots in a row of your choice at 2 minutes per move.

Of course I've already played all of the top bots, so I suppose my chances are already better than 90% in your view.

Edit: minor fix

Title: Re: 2014 State of the Challenge
Post by Fritzlein on Feb 22^nd, 2014, 11:41pm

on 02/22/14 at 22:39:15, browni3141 wrote:

Of course I've already played all of the top bots, so I suppose my chances are already better than 10% in your view.

Indeed I do think you are better than 90% against any current server CC bot, but I notice that you recently lost one of three to bot_sharp, which I count as new since lightvector said he put in significant work on it this year. Those were fast games, true, but how many Elo narrower do you think the gap is at 30s/move than at 120s/move?

Title: Re: 2014 State of the Challenge
Post by browni3141 on Feb 23^rd, 2014, 11:44am

on 02/22/14 at 23:41:20, Fritzlein wrote:

Sharp seems to give me particular difficulty at faster time controls, for some reason. I would estimate that every doubling of the time control favors me by around 150-200 points against sharp. I'm slightly over 50% to win at blitz time control, and probably about 70% at fast. This may not be clearly supported by my game record against the fast versions of sharp as a whole, but if you consider only my recent history it appears roughly accurate. I have won 80% in my last 10 games against Sharp2012Fast

Title: Re: 2014 State of the Challenge
Post by Ail on Feb 23^rd, 2014, 12:49pm

Sharp2012Fast is 297 ELO below the most current version and Leader in the Ladder.