Author |
Topic: Challenge Screening Rules (Read 6273 times) |
|
deep_blue
Forum Guru
Arimaa player #9854
Posts: 212
|
 |
Re: Challenge Screening Rules
« Reply #30 on: Mar 24th, 2015, 9:50am » |
 Quote  Modify
|
I still think that Screening is a good thing and that a much larger WCC should be used that the left lives of the winner count as points for it.
Fritzlein, I seem to have forgot to mention it: My idea was that players with lower games so far have higher priority to play games. That means that one server can only played by players who have less games than some arbitrary number (like e.g. 1). The other server(s) can be used by everyone and like this it's very unlikely that someone with few games doesn't get any more games. (since it's very rare anyway that there is more than one game at the same time)
|
|
 IP Logged |
|
|
|
Janzert
Forum Guru
Arimaa player #247
Gender: 
Posts: 1016
|
|
Re: Challenge Screening Rules
« Reply #31 on: Mar 25th, 2015, 1:40pm » |
Quote Modify
|
To distinguish from a screening period that is used to determine the challenger I'm going to start calling the scenario where the bots are available for play but the challenger is already determined an exhibition period.
on Mar 24th, 2015, 1:28am, Fritzlein wrote:| If individuals are not limited in the number of Screening games they are allowed play, should we worry that a small number of individuals will monopolize the two servers, such that many people won't find the bot ever available at a convenient time? |
|
Maybe, maybe not. If it is a problem, or even just enough concern that it could be a problem, there are a number of straightforward and relatively easy to implement ways to mitigate it. To name a few, cool down periods (static or increasing), a queue, threshold after which the player can only start a game if both servers are currently free.
One important point to note is, should a player circumvent whatever the control is in place, it doesn't in any way threaten the integrity of the exhibition period.
Regarding: Maximum games a bot could be played would be ~500 or ~10 times the highest screening to date.
Quote:| Wait, are you saying that as long as we don't let humans play ten times as many games as before, you aren't worried about moving the goalposts? Say, triple the number of Screening games against the eventual Challenger would not be a problem? (Note that having a single bot available for play on both servers instead of two bots in itself doubles the exposure of that bot.) |
|
(The 2 server setup is accounted for in the above number; 3 weeks * 7 days * 24 hours * 2 servers / 2 hours per game = 504 games)
I'm saying that as a bot developer going from a screening period that has historically had a maximum of ~50 games played against the bot to an exhibition period that could theoretically have up to ~500 games, this is a tradeoff I don't even have to think about. It's an instant yes, please make the change.
One other thing to note is that this proposal is just doubling the theoretical maximum number of games that could be played against the challenger. Of course the current setup would be a bit harder to approach that theoretical limit since it requires the coordination of more than 120 players. 
Three times the number of games in practice would still be an instant yes. Somewhere around five times the number of games actually played I might start needing to actually think about the proposition. 
Quote:| I have always wanted as many different humans as possible to participate in the Screening, but somehow to me that feels less hard on the bots than allowing a few individuals to play over and over and over. Not only does the former make the Screening more of a community event than the latter, allowing the latter would, I think, significantly increase the ability of humans to defend the Challenge due to the efforts of dedicated bot-bashers. |
|
One scenario that has always worried me about the challenge is to have a bot win it and then shortly after that a bot basher finds a simple trick that always beats the bot. Where the trick could be taught to a beginner player easily without needing to understand good Arimaa play in general. The screening was suppose to help mitigate this but I don't think 2 games per human really accomplishes much. The general development of the bots and bot bashing has done more to reassure me. But allowing the bot bashers to really bot bash for a few weeks would be even better.
Quote:| One can make a case that the current exposure of the bots is too little, but it seems clear-cut that to increase it now creates bad publicity. Surely there is a way to change the format to address your concerns about it without also making it harder for the Challenge to be won? |
|
Where does that bad publicity come from? If the group the change is suppose hurt are the primary proponents for it, how can someone say the change was made maliciously against that group?
In reality I doubt any change to the screening is likely to hardly even be noticed outside of the Arimaa community proper. The most detailed accounting of Arimaa happenings I know of, outside of an Arimaa specific site, is the Wikipedia page. Currently the Wikipedia page, while mentioning the change to 3 challenge defenders, doesn't have a single mention of the screening. Which leads to the rather weird situation of having no explanation for why the WCC winner is not always the challenger. I expect someone will probably fix this now that I've mentioned it though.
Janzert
|
|
IP Logged |
|
|
|
Boo
Forum Guru
Arimaa player #6466
Gender:
Posts: 118
|
|
Re: Challenge Screening Rules
« Reply #32 on: Mar 26th, 2015, 2:27am » |
Quote Modify
|
For me it seems strange that not all games are taken into account while determining bot's strength. I am talking about uncompleted pairs. The sample of games is small already, and the current system makes it even smaller.
Maybe awarding 0.5 for being ahead in uncompleted pair or counting total win% would be better solutions.
|
|
IP Logged |
|
|
|
Janzert
Forum Guru
Arimaa player #247
Gender:
Posts: 1016
|
|
Re: Challenge Screening Rules
« Reply #33 on: Apr 1st, 2015, 8:41pm » |
Quote Modify
|
Pulling a related comment out of another thread to reply here.
on Apr 1st, 2015, 1:44am, Fritzlein wrote:| Incidentally, my expectation that a win-on-score formula [against sharp2015] exists even though we haven't quite discovered it yet... |
|
I feel very much the same.
Quote:| ...reinforces my belief that it would be meaningfully unfair to bots to allow individuals unlimited Screening games in which to work out formulaic wins. |
|
Yet have the opposite reaction from it.
For me one of the big reasons to expose the challenge bot to human players pre-challenge is to try and avoid the situation where a bot wins the challenge and then a 'formulaic' method to beat the bot is found shortly thereafter. As a developer having to guard against humans developing bot bashing tricks doesn't bother me because that should be a much easier goal, and really sub-task of the primary goal of developing a bot that plays strategically and tactically superior to humans anyway.
Janzert
|
| « Last Edit: Apr 1st, 2015, 8:43pm by Janzert » |
IP Logged |
|
|
|
Belteshazzar
Forum Guru
Arimaa player #5094
Gender:
Posts: 108
|
|
Re: Challenge Screening Rules
« Reply #34 on: Apr 17th, 2015, 8:03am » |
Quote Modify
|
What about a screening for potential human defenders? Chessandgo seemed like a safe choice, but his losing streak to Sharp2014_Fast would have been a huge red flag had it occurred before he was selected. In the perhaps unlikely event that the Challenge survives past this year, anyone who wants to defend should have to undergo some kind of screening against the previous year's bots (perhaps the Fast versions, to save time) running on the dedicated server. If more than three people pass, the tiebreaker could be how they finish in the WC. If less than three people pass, the other top WC finishers could get the remaining spots.
|
| « Last Edit: Apr 17th, 2015, 1:26pm by Belteshazzar » |
IP Logged |
|
|
|
mattj256
Forum Guru
Arimaa player #8519
Gender:
Posts: 138
|
|
Re: Challenge Screening Rules
« Reply #35 on: Apr 17th, 2015, 2:53pm » |
Quote Modify
|
on Apr 17th, 2015, 8:03am, Belteshazzar wrote:| What about a screening for potential human defenders? |
|
I thought that Omar and Aamir currently choose who will be the challengers each year? I don't have a problem with that because they are putting up most of the prize fund.
I think regardless of the outcome of the tournament, I can't fault them for selecting a six-time world champion as a defender. And if lightvector wins I will tip my cap to him and say that he absolutely deserves the prize. I hope harvestsnow and browni3141 have more success than chessandgo in defending the challenge!
I think it's premature to talk about next year's defenders until we know how Sharp does this year. If browni3141 and harvestsnow both defend the challenge it will be a very different conversation than if Sharp runs the table.
|
| « Last Edit: Apr 17th, 2015, 3:04pm by mattj256 » |
IP Logged |
|
|
|
Belteshazzar
Forum Guru
Arimaa player #5094
Gender:
Posts: 108
|
|
Re: Challenge Screening Rules
« Reply #36 on: Apr 17th, 2015, 5:28pm » |
Quote Modify
|
on Apr 17th, 2015, 2:53pm, mattj256 wrote:| I think it's premature to talk about next year's defenders until we know how Sharp does this year. If browni3141 and harvestsnow both defend the challenge it will be a very different conversation than if Sharp runs the table. |
|
If both defend the challenge, we might forget how close it came. I'd rather not see the challenge fall until a bot is truly unbeatable per the current understanding of Arimaa. If the challenge survives this year, next year we should narrow down the players who are best suited to defend against a super-strong bot. Maybe by then chessandgo will qualify, but this year he wouldn't have.
|
|
IP Logged |
|
|
|
Fritzlein
Forum Guru
Arimaa player #706
Gender:
Posts: 5928
|
|
Re: Challenge Screening Rules
« Reply #37 on: Apr 18th, 2015, 5:43pm » |
Quote Modify
|
Well, now all this discussion is moot. There is no need to worry about moving the goalposts after a goal has been scored. I suppose now the thing to do is simply put the winning bots up for a period of open play after the Computer Championship, while the servers are still being rented, and then immediately make them into regular server bots. Assuming we still have an annual Computer Championship.
|
|
IP Logged |
|
|
|
Belteshazzar
Forum Guru
Arimaa player #5094
Gender:
Posts: 108
|
|
Re: Challenge Screening Rules
« Reply #38 on: Apr 29th, 2015, 8:25pm » |
Quote Modify
|
It looks like this is not moot after all. Does anyone have any thoughts on my idea of screening potential Challenge defenders using the previous year's Fast bots running on the dedicated server?
|
| « Last Edit: Apr 29th, 2015, 8:25pm by Belteshazzar » |
IP Logged |
|
|
|
Belteshazzar
Forum Guru
Arimaa player #5094
Gender:
Posts: 108
|
|
Re: Challenge Screening Rules
« Reply #39 on: May 1st, 2015, 8:58pm » |
Quote Modify
|
All this talk about psychological pressure gave me another idea. If the Challenge really does continue next year, the best defender should go first. This year, that would have been browni. Had he gone first in the first round, he likely would have felt less pressure. Even if he had lost, the other two would have known that browni had a reasonable chance of winning his next two, so his loss wouldn't have put too much additional pressure on them. If all three lose their first game, there will be great pressure in the second round regardless of the order, but if humans start 0-4 including two losses by the best defender, the challenge is probably lost anyway.
|
|
IP Logged |
|
|
|
mattj256
Forum Guru
Arimaa player #8519
Gender:
Posts: 138
|
|
Re: Challenge Screening Rules
« Reply #40 on: May 2nd, 2015, 12:00am » |
Quote Modify
|
on May 1st, 2015, 8:58pm, Belteshazzar wrote:| If the Challenge really does continue next year, the best defender should go first. |
|
Yes that is a good idea and there is real value on minimizing psychological pressure on the strongest human defender. (If you're going to go this route I would say you might as well let the strongest defender choose which position they want.) The sixth game in particular must have been intense, because the challenge really WAS riding on browni's shoulders.
In the current format each round lasts one week, so it takes three weeks to play nine games. If you start reordering games by player strength then you have to either accept a longer tournament or ask the humans to play at times that are less than ideal. The first option is a little bad and the second option is very bad.
|
|
IP Logged |
|
|
|
deep_blue
Forum Guru
Arimaa player #9854
Posts: 212
|
|
Re: Challenge Screening Rules
« Reply #41 on: May 2nd, 2015, 11:33am » |
Quote Modify
|
Now that the Challenge has been won I am not sure if we need the Screening anymore. Alternatively we could do the following:
All players are allowed to play as many games as they want against the top 2 bots. This at the same time IS the challenge. So anyone who plays at least 3 games against the bot has a mini match that counts. If you win 3/5 or 4/7 you of course also won. So we could have the whole community trying to win the challenge back from the bots. Also we would see when even the second best bot beats humanity convincingly.
|
|
IP Logged |
|
|
|
|
Home
Help
Search
Members
Login
Register
Reply
Notify of replies
Send Topic
Print