Welcome, Guest. Please Login or Register.
Mar 28th, 2024, 5:54am

Home Home Help Help Search Search Members Members Login Login Register Register
Arimaa Forum « 2012 Arimaa Challenge »


   Arimaa Forum
   Arimaa
   Events
(Moderator: supersamu)
   2012 Arimaa Challenge
« Previous topic | Next topic »
Pages: 1 2 3 4  Reply Reply Notify of replies Notify of replies Send Topic Send Topic Print Print
   Author  Topic: 2012 Arimaa Challenge  (Read 7837 times)
mistre
Forum Guru
*****





   


Gender: male
Posts: 553
Re: 2012 Arimaa Challenge
« Reply #45 on: Apr 16th, 2012, 12:22pm »
Quote Quote Modify Modify

I think a MORE fair way to do it (and would make more sense) would have been for humans to have to win a majority of the matches in a majority of the pairings.  So, 2 of the 3 challengers would have to go at least 2-1 vs the bot.  The 3rd challenger could still go 0-3, so the bot could still finish with a winning record and lose the challenge (but only 5-4 and not 7-2).
 
If you want to reduce the possibility of a fluke, then make each series best of 5 instead of best of 3.
 
Another way of looking at this is the following scenario:  If Omar had said that their would be 9 human challengers instead of 3, would it make more sense that at least a majority of of the challengers (5 out of 9) would have the win the series or if only one challenger would have to win the series (1 out of 9). If the 2nd scenario is the case, the more people you add, the more you tilt the odds in favor of the humans.
 
But this is Omar's challenge and it is his rules.  He can make it as unfair as he wants it...   Grin
« Last Edit: Apr 16th, 2012, 12:24pm by mistre » IP Logged

Fritzlein
Forum Guru
*****



Arimaa player #706

   
Email

Gender: male
Posts: 5928
Re: 2012 Arimaa Challenge
« Reply #46 on: Apr 16th, 2012, 1:25pm »
Quote Quote Modify Modify

on Apr 16th, 2012, 12:22pm, mistre wrote:
I think a MORE fair way to do it (and would make more sense) would have been for humans to have to win a majority of the matches in a majority of the pairings.  So, 2 of the 3 challengers would have to go at least 2-1 vs the bot.  The 3rd challenger could still go 0-3, so the bot could still finish with a winning record and lose the challenge (but only 5-4 and not 7-2).

That's the scenario for which I ran the math.
 
Quote:
If you want to reduce the possibility of a fluke, then make each series best of 5 instead of best of 3.

But still, a bot could lose three years, win one year, then lose the next three years, right?  How is it fair that the bot can win one even Challenge match in seven years, lose six of them, and still be considered better than all humans?
 
Quote:
But this is Omar's challenge and it is his rules.  He can make it as unfair as he wants it...   Grin

Indeed, our arguing isn't going to change the rules, and it shouldn't.  I was upset when Omar changed the rules in the past, and I will be upset again if he changes the rules in the future.  You lose credibility in a big hurry if you offer a monetary prize and then change what people need to do to win it after they have already put in a bunch of effort.
 
How we talk about the rules does matter, though, even if the rules are set in stone.  I think it is important to understand that the unfairness (which I freely admit) of having to win all three matches instead of just two is balanced by the unfairness (which you haven't acknowledged) that the bot only has to win one year to win for all time, even if the bot isn't yet good enough to win a majority of games in a majority of years.
 
I do hope that we are never in a situation where it looks like a bot is able to win a majority of games from all humans, but humanity keeps defending the Challenge on a fluke.  Humans defending with a 2-7 score would be embarrassing, because it would suggest that even the player who won his match 2-1 might not be better than the bot, and might have merely gotten lucky.
« Last Edit: Apr 16th, 2012, 1:37pm by Fritzlein » IP Logged

mistre
Forum Guru
*****





   


Gender: male
Posts: 553
Re: 2012 Arimaa Challenge
« Reply #47 on: Apr 16th, 2012, 3:47pm »
Quote Quote Modify Modify

on Apr 16th, 2012, 1:25pm, Fritzlein wrote:

That's the scenario for which I ran the math.
 
I do hope that we are never in a situation where it looks like a bot is able to win a majority of games from all humans, but humanity keeps defending the Challenge on a fluke.  Humans defending with a 2-7 score would be embarrassing, because it would suggest that even the player who won his match 2-1 might not be better than the bot, and might have merely gotten lucky.

 
From an earlier post - Btw: if bots have a lower variance in winning (they very consistently beat players below a certain mark, and very consistently lose to players above a different mark) isn't the probability distribution of the performance rating dependent on the strength of opponents in a way that a human's performance rating wouldn't?
 
I know that the comment above was not proven in the screening period, but you stated that you believed the statement and it might have not proven true due to the small sample size of games.  If the statement above is true, does that make your math for which you ran the scenario unreliable?
 
I can't prove my theory, but I see the scenario that a bot losing the challenge with a 2-7 record as more likely to occur at some point in time than the challengers losing by a fluke (all finishing 1-2 even though they are all stronger than the bot).  I guess time will tell.
 
Another point I would like to make - even at the current playing strength of the top bots, there exists the possibility that certain players are just better at botbashing versus playing humans. For example Harren and Max (who went a combined 7-1 in the screening), might have fared better in the challenge than Nombril even though they are ranked lower than Nombril in WHR.  There actually might come a time when the best players for the challenge might not actually be the strongest players vs humans....
IP Logged

Fritzlein
Forum Guru
*****



Arimaa player #706

   
Email

Gender: male
Posts: 5928
Re: 2012 Arimaa Challenge
« Reply #48 on: Apr 16th, 2012, 9:43pm »
Quote Quote Modify Modify

on Apr 16th, 2012, 3:47pm, mistre wrote:
From an earlier post - Btw: if bots have a lower variance in winning (they very consistently beat players below a certain mark, and very consistently lose to players above a different mark) isn't the probability distribution of the performance rating dependent on the strength of opponents in a way that a human's performance rating wouldn't?
 
I know that the comment above was not proven in the screening period, but you stated that you believed the statement and it might have not proven true due to the small sample size of games.  If the statement above is true, does that make your math for which you ran the scenario unreliable?

Yes, if bot vs. human games have a lower variance than human vs. human games, then that throws off my calculation.  That would make any result more indicative of the true man vs. machine relationship.
 
Quote:
I can't prove my theory, but I see the scenario that a bot losing the challenge with a 2-7 record as more likely to occur at some point in time than the challengers losing by a fluke (all finishing 1-2 even though they are all stronger than the bot).  I guess time will tell.

Indeed, it looks like the Challenge is going to be close enough down the wire that it could play out in various flukey ways.  It is looking more likely that the exact rules will matter before 2020.
 
Quote:
Another point I would like to make - even at the current playing strength of the top bots, there exists the possibility that certain players are just better at botbashing versus playing humans. For example Harren and Max (who went a combined 7-1 in the screening), might have fared better in the challenge than Nombril even though they are ranked lower than Nombril in WHR.  There actually might come a time when the best players for the challenge might not actually be the strongest players vs humans....

I agree, although I expect the effect is weaker with a new bot that folks haven't had time to work out a formula for.
« Last Edit: Apr 16th, 2012, 9:45pm by Fritzlein » IP Logged

Arimabuff
Forum Guru
*****



Arimaa player #2764

   


Gender: male
Posts: 589
Re: 2012 Arimaa Challenge
« Reply #49 on: Apr 17th, 2012, 6:52am »
Quote Quote Modify Modify

I think the rules of the challenge should reflect what we are trying to prove with the challenge, for example if we want to make it absolutely certain that a bot dominates humanity we could impose that a bot that wins an equitable challenge against say three humans, would have to play an entire year without change against human players, on this site and then if after that year the bot still wins again under the same equitable conditions against three players then there shouldn't be any doubt that the bot is definitely superior to humanity.
IP Logged
hyperpape
Forum Guru
*****



Arimaa player #7113

   


Gender: male
Posts: 80
Re: 2012 Arimaa Challenge
« Reply #50 on: Apr 17th, 2012, 7:22am »
Quote Quote Modify Modify

While it might be nice for the Arimaa challenge to perfectly reflect the question of who is stronger, it seems more like a signpost. Any bot that can win the challenge is capable of playing at about the same level as the top humans. A judgment of whether it's just close, equal or above them has to depend on the circumstances.  
 
I think Patrick's idea of seeing how the bot fares after open play is a good one. It just doesn't have to be attached to the prize.
IP Logged
hyperpape
Forum Guru
*****



Arimaa player #7113

   


Gender: male
Posts: 80
Re: 2012 Arimaa Challenge
« Reply #51 on: Apr 17th, 2012, 7:39am »
Quote Quote Modify Modify

I've been recording information about the match as it progresses on the wiki, but as far as analysis, I'm limited to reporting what others say. If anyone else wants to add thoughts about the way the games have gone, feel free to do so (or relay them to me if you don't have an account).
IP Logged
Pages: 1 2 3 4  Reply Reply Notify of replies Notify of replies Send Topic Send Topic Print Print

« Previous topic | Next topic »

Arimaa Forum » Powered by YaBB 1 Gold - SP 1.3.1!
YaBB © 2000-2003. All Rights Reserved.