Arimaa Forum - Print Page


    
      
        Arimaa Forum
        (http://arimaa.com/arimaa/forum/cgi/YaBB.cgi)
      

        Arimaa >> Events >> 2015 Arimaa Challenge
        
(Message started by: Fritzlein on Mar 1^st, 2015, 8:45am)

Title: 2015 Arimaa Challenge
Post by Fritzlein on Mar 1^st, 2015, 8:45am

It must be time to start the 2015 Arimaa Challenge thread, because Omar has announced the defenders in game room. They are browni3141, chessandgo, and harvestsnow, with clyring as alternate.

Would anyone care to make predictions for the score this year, as well as for humanity's odds of defending for another five years until the Challenge prize expires?

Title: Re: 2015 Arimaa Challenge
Post by Fritzlein on Mar 1^st, 2015, 8:50am

To get things started, supersamu and I have placed a wager on the screening. If he wins all four of his screening games this year, he wins 200 Arimaa points from me. If he doesn't, I win 100 Arimaa points from him. Right supersamu?

Title: Re: 2015 Arimaa Challenge
Post by supersamu on Mar 1^st, 2015, 9:04am

I first proposed the bet roughly 2 weeks ago, and Fritzlein took the chance for the symbolic amount. The bet translates to me having a 75% winning chance in each game. I am excited what the screening brings, especially now that sharp is appearing so strong (no loss in the CBE).

I am convinced that if you win 2 games in the screening, you have what it takes to get into the finals of the WC. (Meaning losing less than 3 games in the first 6 rounds with the current format).

The screening is also very important for omar and the challenge defenders. Last year we had several newcomers taking a stab and they performed quite well, showing that with basic strategic understanding, you can do surprisingly well.

I hope we have a lot of screening games to show that humanity still reigns over the bots!

Title: Re: 2015 Arimaa Challenge
Post by rbarreira on Mar 1^st, 2015, 10:55am

Another topic that got a short discussion in today's chat was - what time control have the bots already surpassed humans at?

I think almost everyone would agree that humans would have bad winning chances if the challenge was played at 8 seconds per move (lightning).

How about blitz? It's uncertain, but I feel bots may be the best at blitz at the moment, after looking at bot_sharp2014blitz's record.

It would be interesting to find out the time gap between bots getting better at lightning and bots getting better at blitz (if that already happened).

Title: Re: 2015 Arimaa Challenge
Post by aaaa on Mar 1^st, 2015, 2:09pm

At that speed, the interface becomes a big issue.

Title: Re: 2015 Arimaa Challenge
Post by browni3141 on Mar 1^st, 2015, 4:51pm

on 03/01/15 at 10:55:28, rbarreira wrote:

I would definitely doubt my own chances at lightning, but like aaaa says, the interface is important. Moving to the library or anywhere with a low latency connection, for example, is probably worth 100-200 elo. I am therefore not confident in a probable failure, but still skeptical of my chances to score >50%.
At blitz I definitely think I am at least as strong as sharp2014Blitz. When you referenced my record in the chatroom, you included unrated games which were played unrated for very good reasons (most or all were part of simuls). Among rated games only I was 1-2 against sharp, not 1-12, (2-2 as of this posting) In a serious blitz match I could still gain by moving to a faster connection, but not nearly as much as with lightning.

I will happily supply data by playing any bots at blitz/lightning time controls. I will even play games with stakes, but in that case I would need to make sure I can move to a good connection and we would need to agree on odds.

To answer the OP: My very rough estimate of the number of games the bots will win in the Challenge is 2.
As for our chances of defending the Challenge until it expires? That is extremely hard to answer, as it depends on a lot of unknowns. Sharp seems to have improved a lot this year, but it's hard to tell how much lightvector can keep it up or whether new bots will come into the mix, and hard to tell how much humans will improve, etc.
At the current rate of improvement sharp would probably overcome humans, but I don't think this rate of improvement is sustainable. My official answer is "I have no idea whatsoever."

Title: Re: 2015 Arimaa Challenge
Post by Fritzlein on Mar 2^nd, 2015, 2:22pm

on 03/01/15 at 16:51:57, browni3141 wrote:

I will happily supply data by playing any bots at blitz/lightning time controls.

The trouble with using server bots like bot_sharp2014Blitz as a gauge is that they run on a single core, whereas the Challenge is run on twelve cores, and also that server bots are static so they don't have the surprise factor working for them that new bots have in the actual Challenge. (The multi-core and surprise factor issues are the reason that the games of the Screening are my preferred measure of bot strength at CC time controls, even though those ratings are (quite naturally) higher than the gameroom ratings of past CC bots.)

For blitz the most realistic measure would likely be a developer hosting his latest version on his own machine. Unfortunately, any developer trying to win the Challenge in the current year probably wants to expose his bot as little as possible, particularly to one of the actual defenders, but maybe no developer expects to win this year.

Title: Re: 2015 Arimaa Challenge
Post by Fritzlein on Mar 2^nd, 2015, 2:24pm

on 03/01/15 at 16:51:57, browni3141 wrote:

To answer the OP: My very rough estimate of the number of games the bots will win in the Challenge is 2.

For those not reading the entire chat archive, SilverMitt estimated the number of bot wins (assuming the Challenger is bot_sharp) at 3 or 3.5.

Title: Re: 2015 Arimaa Challenge
Post by browni3141 on Mar 2^nd, 2015, 2:52pm

on 03/02/15 at 14:22:53, Fritzlein wrote:

My post was intended as a challenge to latest and greatest bots from the developers. I'll be playing the server bots anyway.

Title: Re: 2015 Arimaa Challenge
Post by deep_blue on Mar 2^nd, 2015, 6:29pm

I claim that the bot (probably sharp) has good chances (>50%) to win one match, but not more. Therefore i agree to 3-3.5 wins by the bot.

Title: Re: 2015 Arimaa Challenge
Post by Fritzlein on Mar 2^nd, 2015, 9:31pm

If I recall correctly from last year, I was predicting about 3.25 bot wins, and bot_ziltoid only managed 2 wins, but I haven't learned from experience. :)

Title: Re: 2015 Arimaa Challenge
Post by Fritzlein on Mar 4^th, 2015, 3:57pm

In the chat room, browni3141 and I negotiated for him to sell me some insurance. I am on the hook for $1000 if the Arimaa Challenge is won this year, and I am worried about having to pay out to lightvector, but browni3141 rates the odds of that happening as very low. (Exactly how low, browni? Less than 100:1 for sure, since you need a premium to justify taking on increased variance.)

In any event, I have sent 200 Arimaa points to browni3141, and he has promised to give me all of his prize money from this year's World Championship if the Challenge is won this year. If he wins the World Championship, that is 23994 Arimaa points of insurance, i.e. almost a quarter of my liability. Thanks, browni!

Title: Re: 2015 Arimaa Challenge
Post by chessandgo on Mar 5^th, 2015, 3:52am

Was it intentional that you didn't take into account in your calculations that other players than browni believe they can win? :)

Title: Re: 2015 Arimaa Challenge
Post by Fritzlein on Mar 5^th, 2015, 12:36pm

on 03/05/15 at 03:52:41, chessandgo wrote:

Was it intentional that you didn't take into account in your calculations that other players than browni believe they can win? :)

I didn't consciously take into account anyone's beliefs other than my own, but I would nevertheless be very interested to hear yours. What do you think your odds of winning are? (game or match or sweep; however you feel like expressing it)

If your confidence rivals browni's, then I would be interested in buying some 100:1 Challenge insurance from you too, indeed as much as you care to offer up to $760. That is to say, I will gladly pay you $7.60 today if you promise to pay me $760 if the Arimaa Challenge is won this year.

Title: Re: 2015 Arimaa Challenge
Post by Fritzlein on Mar 5^th, 2015, 12:48pm

Lifting from the chat room for posterity:

quasar

Quote:

I think Fritzlein made a positive expected value bet.

browni3141

Quote:

If sharp is as good as I think, it's about .1% to win [the 2015 Challenge]. :P

petermck

Quote:

I predict the Arimaa Challenge will be won by 2018.

quasar

Quote:

Arimaa 2020, Go 2024, if I had to guess.

deep_blue

Quote:

arimaa 2035; go 2030 :P

Title: Re: 2015 Arimaa Challenge
Post by Fritzlein on Mar 6^th, 2015, 11:12am

The Computer Championship is over and sharp has won undefeated. The last time a bot was this dominant over its rivals was Bomb in 2007. Sharp's 9-0 performance is even more impressive on the heels of winning the Computer Blitz Event with a 9-0 score as well. Last year in the CBE and CC combined, sharp was 6-6 collectively against its two main rivals, marwin and ziltoid. This year sharp was 13-0 against marwin and Z.

The end of the CC means the beginning of the Screening. Z took a clear second place ahead of marwin, and is thus the second challenger. Below is a table of past screening performances, expressed in terms of game room ratings of the humans who participated in the screening. I will update the table as the event progresses.

Year Pairs Decisive Winner / Score / Perf Loser / Score / Perf
---- ----- -------- --------------------- --------------------
2007 12 . 2 . bomb / 2 / 2087 . Zombie / 0 / 1876
2008 16 . 7 . bomb / 6 / 1918 . sharp / 1 / 1576
2009 23 . 7 clueless / 5 / 1910 . GnoBot / 2 / 1792
2010 25 . 11 marwin / 6 / 2065 clueless / 5 / 1960
2011 40 . 11 marwin / 6 / 2110 . sharp / 5 / 2109
2012 33 . 7 briareus / 5 / 2232 . marwin / 2 / 2128
2013 25 . 6 marwin / 4 / 2121 ziltoid / 2 / 2055
2014 33 . 11 ziltoid / 6 / 2259 . sharp / 5 / 2244
2015 27 . 8 sharp / 8 / 2557 . Z / 0 / 2123

Title: Re: 2015 Arimaa Challenge
Post by Fritzlein on Mar 6^th, 2015, 11:16am

Predicting that sharp will win the Screening is hardly worth the screen space. Is anyone betting on Z? I'll make a bolder prediction to go alongside my checkered past of bold predictions: we will have a shutout for the first time since 2007. That is not to say that Z will win no games; on the contrary, I expect it to have a strong performance. I merely mean that everyone beaten by Z will also be beaten by sharp, so that Z will win zero pairs of games.

Title: Re: 2015 Arimaa Challenge
Post by chessandgo on Mar 6^th, 2015, 11:59am

I meant that browni's expectation in the WC is strictly less than 23994 Arimaa points.

Title: Re: 2015 Arimaa Challenge
Post by Fritzlein on Mar 6^th, 2015, 2:20pm

on 03/06/15 at 11:59:55, chessandgo wrote:

I meant that browni's expectation in the WC is strictly less than 23994 Arimaa points.

Oh, yes I did take that into consideration. Describing my insurance as a 100:1 bet when I have staked 200 Arimaa points would imply that I think browni's expected payout is 20000 Arimaa points. That would be consistent with a 66.7% chance of him winning first place and a 33.3% chance of him winning second place, with a zero chance of finishing lower. That, in turn, would be consistent with neither you nor browni losing to anyone except each other, and you having a 51.34% chance of winning each individual game against him.

I actually think browni has slightly better than even odds in individual games with you, so I might be getting a bit more than 100:1 on my insurance, and I think he has non-negligible chance of losing to someone other than you, which might get me a bit less than 100:1, but what I have actually purchased should be in that ballpark.

Title: Re: 2015 Arimaa Challenge
Post by aaaa on Mar 7^th, 2015, 1:15am

on 03/06/15 at 11:16:57, Fritzlein wrote:

Just to clarify this, are you also predicting no one will even win with different colors against each bot for a 1-1 result?

Title: Re: 2015 Arimaa Challenge
Post by Fritzlein on Mar 7^th, 2015, 1:32am

on 03/07/15 at 01:15:11, aaaa wrote:

Just to clarify this, are you also predicting no one will even win with different colors against each bot for a 1-1 result?

Yep, that's how I have been counting pairs. I'm not offering to bet on it in that formulation, though. :)

Title: Re: 2015 Arimaa Challenge
Post by Fritzlein on Mar 7^th, 2015, 1:34am

In the first evening, the bots won four and lost one for a combined performance rating of 2398. Pretty rough start for humanity. :-/

Title: Re: 2015 Arimaa Challenge
Post by Belteshazzar on Mar 7^th, 2015, 3:17pm

on 03/06/15 at 11:16:57, Fritzlein wrote:

Predicting that sharp will win the Screening is hardly worth the screen space.

Yes, if it doesn't, the screening process will have to be reconsidered. I suppose hanzack-like antics could cause Z to win.

Title: Re: 2015 Arimaa Challenge
Post by Fritzlein on Mar 8^th, 2015, 9:00pm

In the second five games, humanity did a little better, winning two instead of one. Now that each bot has a loss, they can each get a finite performance rating: 2373 for sharp and 2127 for Z. Still no decisive pairs among three completed, but all four unfinished pairs favor sharp.

Title: Re: 2015 Arimaa Challenge
Post by Belteshazzar on Mar 8^th, 2015, 10:13pm

That you beat sharp on your first try makes me think that the Challenge is safe for this year.

Title: Re: 2015 Arimaa Challenge
Post by Fritzlein on Mar 8^th, 2015, 11:07pm

on 03/08/15 at 22:13:01, Belteshazzar wrote:

That you beat sharp on your first try makes me think that the Challenge is safe for this year.

Would you say more than 99% safe, or less than 99% safe? Just so I know whether my insurance purchase was at a fair level or not. ;)

But seriously, if deep_blue can win against both bots by stalling for eight hours, then I'm ready to say the Challenge is 100% safe.

Title: Re: 2015 Arimaa Challenge
Post by clyring on Mar 9^th, 2015, 9:17am

on 03/08/15 at 23:07:17, Fritzlein wrote:

But seriously, if deep_blue can win against both bots by stalling for eight hours, then I'm ready to say the Challenge is 100% safe.

I would say then only that the challenge should be 100% safe, for there is no guarantee that the defenders will be willing to attempt such a strategy until it is too late. :)

Title: Re: 2015 Arimaa Challenge
Post by deep_blue on Mar 9^th, 2015, 1:19pm

on 03/08/15 at 23:07:17, Fritzlein wrote:

But seriously, if deep_blue can win against both bots by stalling for eight hours, then I'm ready to say the Challenge is 100% safe.

Unless computer says no. (this is supposed to be a joke)
I really hope the game gets restarted tomorrow since although the score strategy didn't work I still didn't feel being worse when the connection crashed.

Title: Re: 2015 Arimaa Challenge
Post by Fritzlein on Mar 13^th, 2015, 10:27pm

After a bit of drama, humans won two of the next six games (counting deep_blue's resumed game but not counting 722caasi's early resignation or either of DanielM's games). Congratulations to deep_blue for both of humanity's wins. The bots lose ground in performance rating, because the humans in this batch were lower-rated. Sharp is now at 2253 and Z at 2139.

Six pairs have been completed, and still none are decisive. My expectations for sharp cruising to victory have been thwarted yet again. Moreover, of the four current incomplete pairs, only three now favor sharp.

Title: Re: 2015 Arimaa Challenge
Post by Fritzlein on Mar 14^th, 2015, 10:18pm

Humans win only one of the next five, which gives the bots a boost. Sharp's performance is now at 2321 and Z's is 2160. We have our first decisive pairs out of eight completed: sharp takes a 2-0 lead. However, of the five incomplete pairs, two now favor Z.

Title: Re: 2015 Arimaa Challenge
Post by Fritzlein on Mar 19^th, 2015, 11:06pm

Humanity won two of the most recent five games, and thus gained ground on silicon. Both of the wins, however, were against Z, while two losses were against sharp, so the gap between the bots has widened. Sharp has won all three decisive pairs of the ten completed pairs, with a performance of 2364. Z trails with a performance of 2062 and meager hopes from leading only two of the six incomplete pairs.

Title: Re: 2015 Arimaa Challenge
Post by Fritzlein on Mar 21^st, 2015, 4:49pm

Humanity won only one of the last six, again against Z. The two completed pairs were indecisive, so sharp's lead remains at only 3-0, but sharp leads in seven of the eight incomplete pairs, and 2438 to 2062 in performance rating.

On a personal note, it feels very much like the top bots have drawn even with me, not only because I split my two games with each of them, but also because sharp's Screening performance is currently higher than my game room rating. Of course four games and one player is too little evidence to base anything on. I merely note that in the prolonged man vs. machine struggle, in every domain that silicon conquers, each individual must face his own personal loss, and this feels like mine.

Yes, there is a good chance I will do better next year, but there is also a fair chance that last year was my last year to get a positive score against the bots. We'll see what happens in the rest of the Screening and remaining years of the Challenge.

Title: Re: 2015 Arimaa Challenge
Post by Belteshazzar on Mar 21^st, 2015, 10:46pm

Interesting that no one has yet defeated sharp in a remotely efficient manner. I wonder if the challengers will be able to win in less than 80 turns.

Title: Re: 2015 Arimaa Challenge
Post by browni3141 on Mar 22^nd, 2015, 1:25am

on 03/21/15 at 22:46:03, Belteshazzar wrote:

Interesting that no one has yet defeated sharp in a remotely efficient manner. I wonder if the challengers will be able to win in less than 80 turns.

I would be horrified if all my games took more than 80 turns to either win or lose.
I expect to average about 35 (planning on three wins)

Note that only two players have beaten sharp at all, and one win was in a deliberately inefficient manner, and the other completely indifferent to efficiency.

Title: Re: 2015 Arimaa Challenge
Post by Fritzlein on Mar 25^th, 2015, 2:06am

Another six games are in the books: sharp won two of two and Z won three of four, so both gain in performance rating. Sharp is up to 2472 and Z up to 2082. We're up to fourteen completed pairs, but still only three decisive, and all three in sharp's favor. In ten incomplete pairs, seven favor sharp.

Title: Re: 2015 Arimaa Challenge
Post by Fritzlein on Mar 28^th, 2015, 12:04am

Another prediction garnered from the chat room, this time kzb52:

Quote:

I don't know if anyone's stepped in to be the prophet of doom for the challenge this year. I guess I can fill that role, since I didn't predict anything before now
I say sharp wins 4 games at a minimum, but probably 5
(some plausible results might be 3-0, 1-2, 0-3, or 2-1, 2-1, 0-3, etc. )

Title: Re: 2015 Arimaa Challenge
Post by PerkofBR on Mar 29^th, 2015, 7:19pm

I predict all defenders will win theirs BO3, with sharp getting 2 to 3 wins. ;)

Title: Re: 2015 Arimaa Challenge
Post by Boo on Mar 30^th, 2015, 8:16am

I think humans will barely hold. 2-1, 1-2, 0-3. I hope I am wrong though.

Title: Re: 2015 Arimaa Challenge
Post by Belteshazzar on Mar 30^th, 2015, 4:41pm

I wonder how Fritz would have done in a third game against sharp. When he beat it in his first game, I assumed the challenge was safe.

Title: Re: 2015 Arimaa Challenge
Post by deep_blue on Mar 31^st, 2015, 4:14am

Fritzlein, where are the new performance ratings? ;)

Title: Re: 2015 Arimaa Challenge
Post by Fritzlein on Mar 31^st, 2015, 8:37am

on 03/31/15 at 04:14:36, deep_blue wrote:

Fritzlein, where are the new performance ratings? ;)

Sorry, my weekend Ultimate tournament consumed more than my weekend. Still, I'm glad somebody missed my updates. :) I guess there is a flurry of activity around the deadline; thirteen games since I last updated. But with less than 24 hours to go, I think I'll just wait for the final entry. In the mean time you can well imagine how stratospheric sharp's performance rating has become. As I write, your win-on-score strategy is already bust; good luck winning over the board!

Title: Re: 2015 Arimaa Challenge
Post by Fritzlein on Apr 1^st, 2015, 12:00am

I'm very pleased with the late flurry of Screening games: twenty-one since I last posted six days ago. The final tallies are:
Sharp won 29 and lost 2 for a performance rating of 2557.
Z won 18 and lost 10 for a performance rating of 2123.
There were 27 completed pairs.*
Sharp won all of the 8 decisive pairs.

*(counting Hufflepup's pair and BlakeD's second pair despite the server color glitch; not counting either of DanielM's two games or 722caasi's second game)

Title: Re: 2015 Arimaa Challenge
Post by Fritzlein on Apr 1^st, 2015, 12:28am

Before moving on to my predictions for the future, let me consider a couple of my past predictions.

I predicted (not very confidently) that sharp would win every decisive pair this year. This came true, but not quite as I had envisioned. Since sharp lost only two games, it only had two chances to lose a pair! Although I did foresee sharp's dominance over Z, I didn't anticipate sharp's dominance over all human participants.

I thought supersamu had less than 1/3 chance of sweeping four screening games, and we made a $1:$2 bet to that effect. Supersamu only played one pair and lost both games, so that dollar is clearly mine. (If it is any consolation, supersamu, my expectations for myself were also higher than my performance.)

I bought insurance from browni3141 against the Challenge being won this year at a rate I estimated at 1:100, although I estimated the true odds of the Challenge being won nearer to 1:200 and browni3141 put it around 1:1000. This is turning out to be a great purchase by me, both because browni3141 is now only one win (and two losses) away from the World Championship, so his prize equity has risen above $200, and also because I now put the odds of the Challenge being won closer to 1:20 now that the Screening is over. The value of my insurance is worth over $10 in my current opinion, even though I paid just $2 for it.

The winner in that prediction is quasar, and for exactly the reason that he gave in the chat room: If sharp had turned out to be worse than expected, I stood to lose at most $2, but if sharp turned out to be better than expected (as it has), I could gain much more than $2 of equity. If I was overestimating sharp's strength by 100 elo, it would costs me much less than it would cost browni3141 to underestimate sharp's strength by 100 elo. (In addition to which, he appears to have underestimated by more than that, but that is getting in to my next post.) Uncertainty always skews in the favor of the insurance buyer, and the strength of the improved sharp was uncertain.

Title: Re: 2015 Arimaa Challenge
Post by Fritzlein on Apr 1^st, 2015, 1:44am

Sharp's performance rating of 2557 is truly intimidating. If we pit it against the game room ratings of the defenders, 2512, 2255, and 2235, for browni3141, chessandgo, and harvestsnow respectively, we can calculate that sharp has a 53% chance of winning the Challenge this year! How, then, am I still giving sharp only a 5% chance or so in my above post?

First, I should confess that, as much as I like to calculate sharp's performance rating as the rating that would have predicted a 29-2 showing, this is a biased estimator. One can demonstrate this by simulation: assume sharp's true rating is some fixed value, say 2400, and use that to generate a million screenings against the 31 actual opponents. (of course also assume the screeners' ratings are accurate and the elo formula is true). Average the calculated performance rating across those million simulations, and the average will be higher than 2400.

This upward bias is not merely because a single perfect screening has a performance of infinity and thus makes the average infinity. One can take infinity out of the picture by adding a draw against a 2400 player to each screening, and even so the average performance rating would be over 2400. This is because the screeners are rather weaker than 2400 on average.

Sorry for the math mumbo-jumbo; the upshot is that my performance rating calculation over-reacts to extreme results. In this respect it is rather like humans. :) We now have a very high measurement of sharp's ability, and this time the uncertainty in our measurement skews to the downside.

Secondly, chessandgo is clearly underrated on the game room scale, and browni3141 probably is slightly underrated as well.

Thirdly, I think it is fairly likely that sharp can be beaten on score. Deep_blue did not manage it on two tries, but it is not obvious that he chose the optimal method to lull sharp into near-repetition. His chosen moves might have provoked advances from sharp that other moves would not have. Since nobody else tried, our information is limited. (Incidentally, my expectation that a win-on-score formula exists even though we haven't quite discovered it yet reinforces my belief that it would be meaningfully unfair to bots to allow individuals unlimited Screening games in which to work out formulaic wins.)

Given that shuffling pieces in the opening can accumulate reserve, there is significant upside and negligible downside for Challenge Defenders to attempt to win on score, and to revert to normal Arimaa only if it fails. I don't know whether the Defenders will try this from the outset, or only in desperation, or not even if they are desperate, but this consideration can only increase humanity's chance of defending.

Fourth and finally, HvB games tend to exaggerate differences as measured on an HvH scale. For example, suppose Alice beats Bomb 50% of the time and Alice beats Charlie 70% of the time. You might guess that since Bomb and Alice are equally good, Bomb will beat Charlie 70% of the time too, but I would guess 80% or more. Alice is error-prone in a way that Bomb is not, so Bomb will blunder away fewer games. Similarly if Alice beats Daniel 30% of the time, you might guess that Bomb will too, but I would guess 20% or less, since Alice can adapt to what she perceives Daniel's strategy to be in a way that Bomb can't.

If I am about equal to sharp, then I expect browni3141 and chessandgo to beat sharp with greater probability than they beat me. (Also I note with relief that both are seriously training at the moment.) Admittedly, by the same token, I expect harvestsnow to lose to sharp with greater probability than he would lose to me, but we only need one mini-match victory to defend.

I predict sharp will most likely win four Challenge games this year, with a slight upward bias, i.e. if it isn't four then it is more likely five than three. Before the Screening that would have made me a doomsayer, but in face of sharp's fantastic results, kzb52 and Boo have passed me on pessimism. On the other hand, I'm not the optimist in the room either, with PerkofBR predicting that even harvestsnow will win his mini-match.

Before the Screening, I was going to stick to my prediction from the last few years that bots have a 30% chance of winning the Challenge before it expires in 2020. Given how dramatically sharp has improved this year, and given how lightvector drops hints that there is plenty more for him to optimize before he runs out of ideas, I have to bump my prediction up to 70% that the Challenge is won. Elapsing time is on our side, but the trend in bot strength versus the trend in human strength is against us much more than I anticipated. Hats off to lightvector!

Title: Re: 2015 Arimaa Challenge
Post by Fritzlein on Apr 1^st, 2015, 1:55am

I should also give the 95% confidence interval on sharp's performance rating: 2312 to 2799. That conveys some idea of how much random variation there is in the screening measurement. This is unrelated to the reasons I gave above to like humanity's chances better than actual performance rating would suggest.

Title: Re: 2015 Arimaa Challenge
Post by chessandgo on Apr 1^st, 2015, 4:05am

on 04/01/15 at 01:44:09, Fritzlein wrote:

Secondly, chessandgo is clearly underrated on the game room scale

I can hear soemone coughing, I believe it's Sharp2014Fast.

on 04/01/15 at 01:44:09, Fritzlein wrote:

Before the Screening, I was going to stick to my prediction from the last few years that bots have a 30% chance of winning the Challenge before it expires in 2020. Given how dramatically sharp has improved this year, and given how lightvector drops hints that there is plenty more for him to optimize before he runs out of ideas, I have to bump my prediction up to 70% that the Challenge is won.

Long term, I'm very pessimistic too (maybe even more so), especially with lightvector expecting no slack in sharp's future improvement.

Title: Re: 2015 Arimaa Challenge
Post by deep_blue on Apr 1^st, 2015, 5:23am

When exactly would you guys think the Challenge will be won?
Do you take bets? ;)
Also what odds do you give me if I bet that the Challenge won't be won at all until 2020?
@Fritzlein: I think it's unlikely that Sharp can be beaten on Score. If it's possible then it's extremely difficult. But like I already mentioned in the other thread, when Sharp advances after some shuffling its advances can be exploited by the human to get a very nice position (like I did).

Title: Re: 2015 Arimaa Challenge
Post by Fritzlein on Apr 1^st, 2015, 9:03am

on 04/01/15 at 05:23:30, deep_blue wrote:

I think it's unlikely that Sharp can be beaten on Score. If it's possible then it's extremely difficult.

There is a difference between how difficult it is to work out a winning formula and how difficult it is to execute that formula. I'm going to claim more experience than you at observing how bot-bashing unfolds. We used to have a very active core of players devoted to pushing the limits of humiliating silicon. It was quite amazing how many things we thought were extremely difficult, if not impossible, which after much experimentation became not only possible, but routine. Yet if all the bot-bashers had tried twice and given up, we would never have discovered any of the extreme exploits.

Don't get me wrong; I don't think sharp is naturally passive in anything like the sense that ziltoid is. I merely think that if you get it set up just exactly right, sharp will trade three for three forever without finding a useful fourth step. Based on an off-hand comment from lightvector, I'm quite sure your chat room speculation about sharp turning up aggression around move 30 is wrong. Sharp has no explicit defense against losing on score, so I think it is merely a matter of finding the perfect set of triggers for passivity.

As impressive as sharp's general strength is, it still isn't a learning bot (as far as we know), so whenever we find a weaknesses to exploit, we can fine tune our exploitation of it to greatly exaggerate that weakness. I have too much observation of bots being triggered into idiotic behavior to believe that sharp is immune based only on general playing strength.

I'm not going to bet on the date the Challenge falls, but I would bet 100 Arimaa points (at even odds) that someone will figure out a recipe to beat sharpCC2015 on score within six months of it becoming available for open play. We would have to somehow clarify what "recipe" means, since winning on score just once is not what I'm talking about. How about this: if anyone beats sharpCC2015 on score three games in a row, no matter how long it takes to build up to that victory streak, it proves you are wrong about it being "extremely difficult" to beat sharp on score.

Title: Re: 2015 Arimaa Challenge
Post by deep_blue on Apr 1^st, 2015, 10:31am

on 04/01/15 at 09:03:53, Fritzlein wrote:

I'm not going to bet on the date the Challenge falls, but I would bet 100 Arimaa points (at even odds) that someone will figure out a recipe to beat sharpCC2015 on score within six months of it becoming available for open play. We would have to somehow clarify what "recipe" means, since winning on score just once is not what I'm talking about. How about this: if anyone beats sharpCC2015 on score three games in a row, no matter how long it takes to build up to that victory streak, it proves you are wrong about it being "extremely difficult" to beat sharp on score.

Hm, for one Score win I wouldn't take that bet. But for three in a row? I am not sure.
But for the sake of a bet, okay I take it. Just one thing, could we bet 1000 Arimaa points? (because then if I win I could enter next year's WC or Postal Mixer ;) ; alternatively: I bet one entry fee of next years WC or Postal Mixer)

Title: Re: 2015 Arimaa Challenge
Post by Fritzlein on Apr 1^st, 2015, 12:54pm

on 04/01/15 at 10:31:30, deep_blue wrote:

No thanks on the larger bet. $1 is a good symbolic amount when I am not trying to earn a profit or insure against loss so much as I am trying to make a point. I don't actually like to gamble: I just like to convert theoretical discussions into something more tangible.

It occurs to me that I will surely have to win the three straight myself even if someone else figures it out, because no sane person would waste all that time applying a reliable formula. Indeed, I may have to work out the formula myself, because you and browni3141 seem to be the only ones interested in winning on score. You aren't likely to win my dollar for me, and browni3141 is mostly trying to provide fuel for changing a rule he doesn't like. Therefore I could easily lose this dollar due to collective apathy. There are much more interesting theoretical questions and more productive challenges in the realm of bot-bashing than trying to induce passivity in a bot.

Maybe if the roles were reversed, i.e. if you were the one who would have to do work to win the bet, then increasing the stakes would seem reasonable. Then I might not mind losing $10; I could see it as a wage I have paid out to advance the frontiers of human knowledge. Indeed there have been cash prizes for past bot-bashing feats.

Would you mind if I call off our bet entirely, and instead offer a $10 bounty for three consecutive score wins against sharp2015CC?

Title: Re: 2015 Arimaa Challenge
Post by deep_blue on Apr 1^st, 2015, 1:36pm

on 04/01/15 at 12:54:53, Fritzlein wrote:

...because no sane person would waste all that time applying a reliable formula...
Would you mind if I call off our bet entirely, and instead offer a $10 bounty for three consecutive score wins against sharp2015CC?

I guess that would mean an insane person like me could win those $10 without risking to lose anything (since it's no bet anymore)? Well, I think it's very unlikely to manage it but if I have nothing to lose anyway, why not?
I take your offer. How much time do I get? Forever? Till next Screening?

Title: Re: 2015 Arimaa Challenge
Post by Fritzlein on Apr 2^nd, 2015, 12:05am

on 04/01/15 at 13:36:49, deep_blue wrote:

I take your offer. How much time do I get? Forever? Till next Screening?

I was thinking six months from sharp2015CC becoming permanently available as a server bot (i.e. the clock doesn't start ticking if Omar merely leaves sharp up for open play through April on the Challenge servers and then doesn't make it available as a regular server bot).

And yes, you can win the $10 (actually 1000 Arimaa points) with no risk of losing anything (except your time :P).

And I want to make the offer open to anyone as long as nobody has figured it out. I somehow want to avoid someone winning two in a row on score and not having time to complete the set, and someone else jumping in to execute their formula three times before they get back. Not sure how to formulate this, but the points got to whoever was first to completely figure it out.

Title: Re: 2015 Arimaa Challenge
Post by Fritzlein on Apr 10^th, 2015, 11:20pm

on 04/01/15 at 00:28:27, Fritzlein wrote:

I now put the odds of the Challenge being won closer to 1:20 now that the Screening is over.

Apparently I didn't give enough weight to sharp's excellent performance in the Screening. With sharp beating chessandgo and harvestsnow in the first two games, I now have to give it a 20% chance of winning the Challenge this year. I still have faith in browni3141. Not as much faith as he has in himself, mind you (or equivalently more respect for sharp), but I expect he is still better than sharp, and the chance that either chessandgo or harvestsnow come back to win two straight is non-negligible. This year's Challenge is way more exciting than it was supposed to be. No matter what happens next, I tip my hat to lightvector for what he has already achieved.

Title: Re: 2015 Arimaa Challenge
Post by Fritzlein on Apr 11^th, 2015, 9:28pm

Wow, with sharp sweeping the first round, humanity is in a tough spot. Yes, we have three people who could each still defend with no help from the other two, keeping the Challenge intact with a 2-7 score. On the other hand, if we now have to conclude that sharp is as good as browni3141, there is less than 1/3 chance of any of these three human comebacks occurring, i.e. sharp is at least 2/3 chance of take at least one more game from each defender, thus winning the Challenge this year. I have apparently been rather wrong about humanity's chance, and clever only insofar as I bought insurance for $2 which is now worth over $160. :-X

Title: Re: 2015 Arimaa Challenge
Post by mattj256 on Apr 12^th, 2015, 3:43am

on 04/11/15 at 21:28:26, Fritzlein wrote:

sharp is at least 2/3 chance of take at least one more game from each defender, thus winning the Challenge this year.

Let's pretend Sharp is 50% to win every game. The odds of Sharp winning the series against an individual defender would be 3/4 or 75%. The odds of Sharp winning all three series would be (3/4)^3, which is 42%.

These statistics are "for entertainment purposes only" and don't make any attempt to address psychological factors. In previous tournaments humanity had already locked up the defense by the 4th or 5th game. In this tournament humanity can't clinch the defense earlier than the 7th game.

I remember two years ago when Browni3141 won his challenge game with a horse handicap. What has changed? Are the bots so much faster and/or better than they were last year?

Matthew

Title: Re: 2015 Arimaa Challenge
Post by rbarreira on Apr 12^th, 2015, 10:38am

on 04/12/15 at 03:43:38, mattj256 wrote:

I remember two years ago when Browni3141 won his challenge game with a horse handicap. What has changed? Are the bots so much faster and/or better than they were last year?

It's too early to give a full summary of what has changed, since we don't have enough data on sharp2015 yet. But lightvector has stated that sharp2015 vs sharp2014 got a 400 elo boost in self-play, which is quite significant. It also appears from the WCC that this boost translated well to dominance over other bots, and the screening and first round of the challenge show it may have translated to a big boost vs humans as well, again compared to the 2014 version of sharp.

In terms of hardware, sharp also got a nice boost - I recall that last year a mistake in sharp's setup meant it was only using half the cores of the 2014 hardware. On top of fixing this, the 2015 version is also running on a CPU that is about ~~43%~~ 25% faster (same CPU architecture with 25% higher clock rate). So this translates to a ~2.5x speedup on top of the software improvements.

edit - corrected the clock rate (the page on 2015 hardware (http://arimaa.com/arimaa/challenge/2015/hardware.html) seems wrong, the X5650 is clocked at 2.66 GHz (http://ark.intel.com/products/47922/Intel-Xeon-Processor-X5650-12M-Cache-2_66-GHz-6_40-GTs-Intel-QPI) and not 3.06 GHz)

Title: Re: 2015 Arimaa Challenge
Post by Fritzlein on Apr 18^th, 2015, 5:09pm

Now that sharp has swept the first six games of the Challenge, insuring victory, let me be the first to post: Congratulations, lightvector! What a phenomenal leap in strength for sharp from 2014 to 2015. I used to think I would be heartbroken to pay out the $1000 I pledged to the prize fund, but now that it comes time, I realize what an honor it is for me to get to give the prize to you. The Arimaa Challenge couldn't have been won by a smarter, nicer, more creative guy. I have a strong hunch that winning the Arimaa Challenge isn't the last thing you are going to be famous for in your life. Rock on!

Title: Re: 2015 Arimaa Challenge
Post by jdb on Apr 19^th, 2015, 8:57am

Well done, lightvector!

Title: Re: 2015 Arimaa Challenge
Post by lightvector on Apr 19^th, 2015, 6:54pm

Thanks Fritzlein and jdb!

Title: Re: 2015 Arimaa Challenge
Post by RonWeasley on Apr 26^th, 2015, 7:52pm

I've been away from the game for a while. I really didn't think the challenge would fall this year, but I saw that good players were not winning easily against the top bots. I'm really surprised to see that the Challenge has been won. Congratulations, lightvector! I'll be looking at the games and looking for your name in the Daily Prophet.

Title: Re: 2015 Arimaa Challenge
Post by Lion on Jan 10^th, 2018, 12:25pm

on 04/02/15 at 00:05:34, Fritzlein wrote:

Is the "Fritzlein challenge" still available? Sharp_2015CC only got uploaded recently (and I feel I have a fairly decent shot at winning against it on score 3 times in a row).

Title: Re: 2015 Arimaa Challenge
Post by omar on Feb 26^th, 2018, 5:52pm

You can try sending Fritz a message and see what he says.