Welcome, Guest. Please Login or Register.
Jan 27th, 2020, 10:17pm

Home Home Help Help Search Search Members Members Login Login Register Register
Arimaa Forum « Challenge Screening Rules »


   Arimaa Forum
   Arimaa
   Events
(Moderator: supersamu)
   Challenge Screening Rules
« Previous topic | Next topic »
Pages: 1 2 3  Reply Reply Notify of replies Notify of replies Send Topic Send Topic Print Print
   Author  Topic: Challenge Screening Rules  (Read 3602 times)
deep_blue
Forum Guru
*****



Arimaa player #9854

   


Posts: 212
Re: Challenge Screening Rules
« Reply #30 on: Mar 24th, 2015, 9:50am »
Quote Quote Modify Modify

I still think that Screening is a good thing and that a much larger WCC should be used that the left lives of the winner count as points for it.
Fritzlein, I seem to have forgot to mention it: My idea was that players with lower games so far have higher priority to play games. That means that one server can only played by players who have less games than some arbitrary number (like e.g. 1). The other server(s) can be used by everyone and like this it's very unlikely that someone with few games doesn't get any more games. (since it's very rare anyway that there is more than one game at the same time)
IP Logged
Janzert
Forum Guru
*****



Arimaa player #247

   


Gender: male
Posts: 1014
Re: Challenge Screening Rules
« Reply #31 on: Mar 25th, 2015, 1:40pm »
Quote Quote Modify Modify

To distinguish from a screening period that is used to determine the challenger I'm going to start calling the scenario where the bots are available for play but the challenger is already determined an exhibition period.
 
on Mar 24th, 2015, 1:28am, Fritzlein wrote:
If individuals are not limited in the number of Screening games they are allowed play, should we worry that a small number of individuals will monopolize the two servers, such that many people won't find the bot ever available at a convenient time?

 
Maybe, maybe not. If it is a problem, or even just enough concern that it could be a problem, there are a number of straightforward and relatively easy to implement ways to mitigate it. To name a few, cool down periods (static or increasing), a queue, threshold after which the player can only start a game if both servers are currently free.
 
One important point to note is, should a player circumvent whatever the control is in place, it doesn't in any way threaten the integrity of the exhibition period.
 
Regarding: Maximum games a bot could be played would be ~500 or ~10 times the highest screening to date.
Quote:
Wait, are you saying that as long as we don't let humans play ten times as many games as before, you aren't worried about moving the goalposts?  Say, triple the number of Screening games against the eventual Challenger would not be a problem?  (Note that having a single bot available for play on both servers instead of two bots in itself doubles the exposure of that bot.)

 
(The 2 server setup is accounted for in the above number; 3 weeks * 7 days * 24 hours * 2 servers / 2 hours per game = 504 games)
 
I'm saying that as a bot developer going from a screening period that has historically had a maximum of ~50 games played against the bot to an exhibition period that could theoretically have up to ~500 games, this is a tradeoff I don't even have to think about. It's an instant yes, please make the change.
 
One other thing to note is that this proposal is just doubling the theoretical maximum number of games that could be played against the challenger. Of course the current setup would be a bit harder to approach that theoretical limit since it requires the coordination of more than 120 players. Smiley
 
Three times the number of games in practice would still be an instant yes. Somewhere around five times the number of games actually played I might start needing to actually think about the proposition. Wink
 
Quote:
I have always wanted as many different humans as possible to participate in the Screening, but somehow to me that feels less hard on the bots than allowing a few individuals to play over and over and over.  Not only does the former make the Screening more of a community event than the latter, allowing the latter would, I think, significantly increase the ability of humans to defend the Challenge due to the efforts of dedicated bot-bashers.

 
One scenario that has always worried me about the challenge is to have a bot win it and then shortly after that a bot basher finds a simple trick that always beats the bot. Where the trick could be taught to a beginner player easily without needing to understand good Arimaa play in general. The screening was suppose to help mitigate this but I don't think 2 games per human really accomplishes much. The general development of the bots and bot bashing has done more to reassure me. But allowing the bot bashers to really bot bash for a few weeks would be even better.
 
Quote:
One can make a case that the current exposure of the bots is too little, but it seems clear-cut that to increase it now creates bad publicity.  Surely there is a way to change the format to address your concerns about it without also making it harder for the Challenge to be won?

 
Where does that bad publicity come from? If the group the change is suppose hurt are the primary proponents for it, how can someone say the change was made maliciously against that group?
 
In reality I doubt any change to the screening is likely to hardly even be noticed outside of the Arimaa community proper. The most detailed accounting of Arimaa happenings I know of, outside of an Arimaa specific site, is the Wikipedia page. Currently the Wikipedia page, while mentioning the change to 3 challenge defenders, doesn't have a single mention of the screening. Which leads to the rather weird situation of having no explanation for why the WCC winner is not always the challenger. I expect someone will probably fix this now that I've mentioned it though. Smiley
 
Janzert
IP Logged
Boo
Forum Guru
*****



Arimaa player #6466

   


Gender: male
Posts: 118
Re: Challenge Screening Rules
« Reply #32 on: Mar 26th, 2015, 2:27am »
Quote Quote Modify Modify

For me it seems strange that not all games are taken into account while determining bot's strength. I am talking about uncompleted pairs. The sample of games is small already, and the current system makes it even smaller.  
Maybe awarding 0.5 for being ahead in uncompleted pair or counting total win% would be better solutions.
IP Logged

Janzert
Forum Guru
*****



Arimaa player #247

   


Gender: male
Posts: 1014
Re: Challenge Screening Rules
« Reply #33 on: Apr 1st, 2015, 8:41pm »
Quote Quote Modify Modify

Pulling a related comment out of another thread to reply here.
 
on Apr 1st, 2015, 1:44am, Fritzlein wrote:
Incidentally, my expectation that a win-on-score formula [against sharp2015] exists even though we haven't quite discovered it yet...

 
I feel very much the same.
 
Quote:
...reinforces my belief that it would be meaningfully unfair to bots to allow individuals unlimited Screening games in which to work out formulaic wins.

 
Yet have the opposite reaction from it. Wink
 
For me one of the big reasons to expose the challenge bot to human players pre-challenge is to try and avoid the situation where a bot wins the challenge and then a 'formulaic' method to beat the bot is found shortly thereafter. As a developer having to guard against humans developing bot bashing tricks doesn't bother me because that should be a much easier goal, and really sub-task of the primary goal of developing a bot that plays strategically and tactically superior to humans anyway.
 
Janzert
« Last Edit: Apr 1st, 2015, 8:43pm by Janzert » IP Logged
Belteshazzar
Forum Guru
*****



Arimaa player #5094

   


Gender: male
Posts: 105
Re: Challenge Screening Rules
« Reply #34 on: Apr 17th, 2015, 8:03am »
Quote Quote Modify Modify

What about a screening for potential human defenders?  Chessandgo seemed like a safe choice, but his losing streak to Sharp2014_Fast would have been a huge red flag had it occurred before he was selected.  In the perhaps unlikely event that the Challenge survives past this year, anyone who wants to defend should have to undergo some kind of screening against the previous year's bots (perhaps the Fast versions, to save time) running on the dedicated server.  If more than three people pass, the tiebreaker could be how they finish in the WC.  If less than three people pass, the other top WC finishers could get the remaining spots.          
« Last Edit: Apr 17th, 2015, 1:26pm by Belteshazzar » IP Logged
mattj256
Forum Guru
*****



Arimaa player #8519

   


Gender: male
Posts: 138
Re: Challenge Screening Rules
« Reply #35 on: Apr 17th, 2015, 2:53pm »
Quote Quote Modify Modify

on Apr 17th, 2015, 8:03am, Belteshazzar wrote:
What about a screening for potential human defenders?

I thought that Omar and Aamir currently choose who will be the challengers each year?  I don't have a problem with that because they are putting up most of the prize fund.  
 
I think regardless of the outcome of the tournament, I can't fault them for selecting a six-time world champion as a defender.  And if lightvector wins I will tip my cap to him and say that he absolutely deserves the prize.  I hope harvestsnow and browni3141 have more success than chessandgo in defending the challenge!
 
I think it's premature to talk about next year's defenders until we know how Sharp does this year.  If browni3141 and harvestsnow both defend the challenge it will be a very different conversation than if Sharp runs the table.
« Last Edit: Apr 17th, 2015, 3:04pm by mattj256 » IP Logged
Belteshazzar
Forum Guru
*****



Arimaa player #5094

   


Gender: male
Posts: 105
Re: Challenge Screening Rules
« Reply #36 on: Apr 17th, 2015, 5:28pm »
Quote Quote Modify Modify

on Apr 17th, 2015, 2:53pm, mattj256 wrote:
I think it's premature to talk about next year's defenders until we know how Sharp does this year.  If browni3141 and harvestsnow both defend the challenge it will be a very different conversation than if Sharp runs the table.

 
If both defend the challenge, we might forget how close it came.  I'd rather not see the challenge fall until a bot is truly unbeatable per the current understanding of Arimaa.  If the challenge survives this year, next year we should narrow down the players who are best suited to defend against a super-strong bot.  Maybe by then chessandgo will qualify, but this year he wouldn't have.    
IP Logged
Fritzlein
Forum Guru
*****



Arimaa player #706

   
Email

Gender: male
Posts: 5928
Re: Challenge Screening Rules
« Reply #37 on: Apr 18th, 2015, 5:43pm »
Quote Quote Modify Modify

Well, now all this discussion is moot.  There is no need to worry about moving the goalposts after a goal has been scored.  Smiley  I suppose now the thing to do is simply put the winning bots up for a period of open play after the Computer Championship, while the servers are still being rented, and then immediately make them into regular server bots.  Assuming we still have an annual Computer Championship.
IP Logged

Belteshazzar
Forum Guru
*****



Arimaa player #5094

   


Gender: male
Posts: 105
Re: Challenge Screening Rules
« Reply #38 on: Apr 29th, 2015, 8:25pm »
Quote Quote Modify Modify

It looks like this is not moot after all.  Does anyone have any thoughts on my idea of screening potential Challenge defenders using the previous year's Fast bots running on the dedicated server?
« Last Edit: Apr 29th, 2015, 8:25pm by Belteshazzar » IP Logged
Belteshazzar
Forum Guru
*****



Arimaa player #5094

   


Gender: male
Posts: 105
Re: Challenge Screening Rules
« Reply #39 on: May 1st, 2015, 8:58pm »
Quote Quote Modify Modify

All this talk about psychological pressure gave me another idea.  If the Challenge really does continue next year, the best defender should go first.  This year, that would have been browni.  Had he gone first in the first round, he likely would have felt less pressure.  Even if he had lost, the other two would have known that browni had a reasonable chance of winning his next two, so his loss wouldn't have put too much additional pressure on them.  If all three lose their first game, there will be great pressure in the second round regardless of the order, but if humans start 0-4 including two losses by the best defender, the challenge is probably lost anyway.
IP Logged
mattj256
Forum Guru
*****



Arimaa player #8519

   


Gender: male
Posts: 138
Re: Challenge Screening Rules
« Reply #40 on: May 2nd, 2015, 12:00am »
Quote Quote Modify Modify

on May 1st, 2015, 8:58pm, Belteshazzar wrote:
If the Challenge really does continue next year, the best defender should go first.
Yes that is a good idea and there is real value on minimizing psychological pressure on the strongest human defender.  (If you're going to go this route I would say you might as well let the strongest defender choose which position they want.)  The sixth game in particular must have been intense, because the challenge really WAS riding on browni's shoulders.
 
In the current format each round lasts one week, so it takes three weeks to play nine games.  If you start reordering games by player strength then you have to either accept a longer tournament or ask the humans to play at times that are less than ideal.  The first option is a little bad and the second option is very bad.
IP Logged
deep_blue
Forum Guru
*****



Arimaa player #9854

   


Posts: 212
Re: Challenge Screening Rules
« Reply #41 on: May 2nd, 2015, 11:33am »
Quote Quote Modify Modify

Now that the Challenge has been won I am not sure if we need the Screening anymore. Alternatively we could do the following:
All players are allowed to play as many games as they want against the top 2 bots. This at the same time IS the challenge. So anyone who plays at least 3 games against the bot has a mini match that counts. If you win 3/5 or 4/7 you of course also won. So we could have the whole community trying to win the challenge back from the bots. Also we would see when even the second best bot beats humanity convincingly.
IP Logged
Pages: 1 2 3  Reply Reply Notify of replies Notify of replies Send Topic Send Topic Print Print

« Previous topic | Next topic »

Arimaa Forum » Powered by YaBB 1 Gold - SP 1.3.1!
YaBB © 2000-2003. All Rights Reserved.