Author |
Topic: Easter eggs and gap between humans and bots (Read 1507 times) |
|
Swynndla
Forum Guru
Arimaa player #1821
Posts: 235
|
 |
Easter eggs and gap between humans and bots
« on: Apr 18th, 2011, 8:09pm » |
 Quote  Modify
|
I heard Fritz say in his last game against marwin that bots in the arimaa champs vs humans have to play the same as in the qualifying rounds (the commentators were wondering if there was an "easter egg" in marwin's program that made it do that bait & tackle opening). Does that mean that bots aren't allowed easter eggs? And when we say that they have to play the same as the qualifying round, is that the qualifying round against humans, or the bot champs? Is the bot's program to remain untouched when it's submitted to the bot champs, or can it be changed after the bot champs and before the human qualifying rounds. I'm just trying to clarify how much a bot can surprise its human opponents, and still be following the rules.
Also I wanted to ask ... I know both humans and bots have come a long way over the years (ie both parties have improved). If marwin had have been around in 2006 (when three humans first defended the title), would marwin have been victorious against humans? And what about in 2004 versus Omar? I just want a feel of how far behind bots are.
|
|
 IP Logged |
|
|
|
Fritzlein
Forum Guru
Arimaa player #706
Gender: 
Posts: 5928
|
|
Re: Easter eggs and gap between humans and bots
« Reply #1 on: Apr 18th, 2011, 10:41pm » |
Quote Modify
|
Your question about the Arimaa Challenge rules is a good one. I don't think the current rules are as good as they need to be.
Omar has explicitly said that he wants to allow a bot to play differently depending on the opponent. It is legal for a bot to be coded to play a certain opening only against Fritz, and a different opening against everyone else. This means it is certainly legal for a bot to play differently in the Computer Championship than in the Arimaa Challenge, based on its knowledge of who it is playing against.
On the other hand, Omar has also explicitly said that he wants a bot to play the same way in the Screening as in the Challenge. Part of the purpose of the Screening is to give humans a chance to probe for weaknesses, and ensure that the bot isn't a complete unknown entering the Challenge.
As far as I can tell, neither of these principles is part of the official rules, which is a problem. Furthermore, the two principles come into conflict somewhat in ways that should be clarified before it becomes an issue. Presumably it is not OK to play differently based on the date of the game, as that would be a clear evasion of Omar's intention. But is it OK to play weak setups against anyone rated under 2300 and strong setups against anyone rated over 2300? That could be viewed as basing play on the opponent, not on the event, and thus be legal, or it could be viewed as trying to conceal the strength of the bot in the Screening so that no winning patterns can be discovered that are useful in the Challenge. We need to think about this more, and definitely get something written into the rules of the Challenge.
I believe that the top bots of today would have been favorites to win against either Omar in 2004 or Belbo in 2005. Omar might have the upper hand on marwin today, but he also knows a lot more than he did back then. I don't think present-day marwin would have won in 2006 or later, partly because the level of human play had risen, and partly because the rules changed to make it much harder on the bots; defeating each of three humans in mini-matches is a far lower probability than defeating a single human in a longer match. Either PMertens or Adanac or I would have come through for humanity, I believe.
|
|
IP Logged |
|
|
|
Swynndla
Forum Guru
Arimaa player #1821
Posts: 235
|
|
Re: Easter eggs and gap between humans and bots
« Reply #2 on: Apr 19th, 2011, 12:26am » |
Quote Modify
|
Thank you Fritz for your well thought-out & informative reply! If the bots of today couldn't beat the best humans five years ago, they really have got quite a way to go if they are to be able to beat top humans by 2020. I suppose humans could plateau, while bots continue to improve due to processing speed increases? I'm not sure if there are any signs of humans plateauing though, but who knows. Is the gap decreasing or increasing I wonder? It seems to me that humans are increasing their lead??
Another side of me is very impressed that the gap isn't that wide, and that humans have had to take the challenge seriously or risk losing. Arimaa seems to be just complex enough (in terms of tree searches) to make the challenge exiting (well I know if I were defending the challenge, the bots would win).
|
|
IP Logged |
|
|
|
Fritzlein
Forum Guru
Arimaa player #706
Gender:
Posts: 5928
|
|
Re: Easter eggs and gap between humans and bots
« Reply #3 on: Apr 19th, 2011, 1:44am » |
Quote Modify
|
Humans certainly increased their lead every year from 2004 to 2008. The last three years are less clear. By game room ratings the top human's lead over the top bot has dipped from about 550 to 450. But game room ratings have tended to understate the rating of the top players ever since the top players stopped bot-bashing (as I used to do). Chessandgo's dominance in WHR is greater, and he seldom plays bots, so the true gap may be larger than his game room rating reflects. Also judging by the size of the human population able to defeat the top bots, it seems we are at least holding steady.
Progress on both the human side and the bot side appears to come in discrete leaps rather than continuously, even if the curve is relatively smooth over a longer time period. If a new player passed chessandgo in strength, that would instantly change the equation. If David Wu comes up with another great idea for sharp, that could upset the apple cart as well. It seems plausible that the gap next year will be smaller than it is this year, but also plausible that it will be larger. We tend to read too much into short-term trends. Unfortunately, it is only in hindsight that the long-term trend becomes clear.
As for Arimaa being just hard enough for computers to make the Challenge exciting, that isn't true on a yearly basis, but is definitely true overall. When I first got to know Omar, I pressed him as to whether he thought he would have to pay out his $10000 by 2020. He said it was about 50-50. Every time that I have asked him since then, including last month, he has said it is about 50-50. That implies a low probability of computers winning next year, or in any particular year, but is as uncertain as it gets overall. Also it implies that Omar thinks the gap is shrinking now or will shrink in the future. If computers make progress just enough faster than humans make progress, it could come down to the final match in 2020 with Omar still saying it is about 50-50!
|
|
IP Logged |
|
|
|
omar
Forum Guru
Arimaa player #2
Gender:
Posts: 1003
|
|
Re: Easter eggs and gap between humans and bots
« Reply #4 on: Apr 20th, 2011, 3:54pm » |
Quote Modify
|
I'll have to look at what I said and see if I still think it is a valid concern. When there is little prior experience one tends to be concerned about issues that may not even arise much in practice. Basically I want the submitted bots to play at full strength at all times and not reduce their play strength based on opponent, event or other factors.
I think the screening period does a pretty good job of requiring the bots to play their best. The competition to score higher during the screening has done a good job of requiring the bots to play their best. But what if a bot was so ahead of even the next best bot that it could afford to not play full strength during the screening period (for example using just one core; or limiting it's search depth; or using weaker setups), but then played full strength in the challenge. This would be considered purposely deceiving.
A few things make this a complex issue.
1. It is not easy to tell if a bot was playing less than it's full strength prior to the challenge match; without requiring access to the original source code.
2. I consider it OK for a bot to play differently against different opponents.
3. I consider it OK for a bot to play different moves from the exact same game state.
It would be extremely hard to enforce that a bot developer not add any code to have the bot play weaker under some conditions. But as long as there is sufficient competition among the bots to make it to the challenge match I think that itself should be enough to enforce it.
|
|
IP Logged |
|
|
|
RonWeasley
Forum Guru
Harry's friend (Arimaa player #441)
Gender:
Posts: 882
|
|
Re: Easter eggs and gap between humans and bots
« Reply #5 on: Apr 20th, 2011, 4:24pm » |
Quote Modify
|
Under the current rules, if we discovered that a bot had done this, as TD I would NOT disqualify the bot or the games played at less than full strength. I believe this level of gamesmanship is permissible, although difficult to achieve as Omar says. I recommend either amending the rules to prevent this (good luck) or being prepared for it to happen if the bots get really smart.
Imagine if a separate set of special human players were treated in the Challenge as the bots are now. The special human challenger could try to do this and I don't see how to stop him.
In the case of the bots, you might consider this an unintended dimension in AI that may be a component of the revolutionary approach you are looking for.
|
|
IP Logged |
|
|
|
rbarreira
Forum Guru
Arimaa player #1621
Gender:
Posts: 605
|
|
Re: Easter eggs and gap between humans and bots
« Reply #6 on: Apr 20th, 2011, 5:04pm » |
Quote Modify
|
Good post omar, and I agree that the current rules already give a big incentive for the bots to play at full strength during the CC and screening.
Just a minor comment on one of your points:
on Apr 20th, 2011, 3:54pm, omar wrote:
3. I consider it OK for a bot to play different moves from the exact same game state.
|
|
Actually with timing issues and multi-threading, it is almost impossible for the bot to choose the same move from a position every time. My bot currently has explicit randomness, but even before it had that, it already chose different moves due to timing/multi-threading alone...
|
| « Last Edit: Apr 20th, 2011, 5:05pm by rbarreira » |
IP Logged |
|
|
|
Swynndla
Forum Guru
Arimaa player #1821
Posts: 235
|
|
Re: Easter eggs and gap between humans and bots
« Reply #7 on: Apr 20th, 2011, 7:38pm » |
Quote Modify
|
on Apr 20th, 2011, 3:54pm, omar wrote:| 2. I consider it OK for a bot to play differently against different opponents. |
|
I'm not really sure what this means, as it could be interpreted that it's ok for a bot to play weaker against other bots than it does for the Challenge (but this isn't what you mean from what you've said above, because that strategy would be deceiving). I'm thinking (from what you've said) you mean that if a bot was programmed to play conservatively against Fritz, and aggressively against chessandgo (or the other way around) then that would be ok. The bot would not be deceiving but would simply be choosing to play different styles base on the type of player it's playing.
As you say, it only becomes a concern if a bot improved dramatically over all the other bots. If I have thought of a break-through-idea for a bot (I haven't, but lets pretend I have) ... let's say that I'd figured out that a bot can win against humans if it plays strategy H (for Human): plays ultra-conservatively against humans, plays like this for hours, and waits for the human to do a blunder, and then pounces on it, while against other bots this strategy H doesn't work, and would instead play strategy B (for Bot): play for positional advantages (again I don't think this strategy would actually work, but again, let's pretend). Then I'm still thinking from what you've said that it'd be okay for a bot to be programmed like that? But what wouldn't be ok is if that bot was programmed to play against humans using strategy B in the qualifying rounds but then after a certain date (knowing that the qualifying rounds are over and the Challenge had started) suddenly changing to strategy H, as that would be deceiving.
Another example ... if I have programmed a bot (I haven't, but I intend to start one day), and I thought it was stronger than all the other bots except one, I could program a strategy into the bot to deal with only that good bot (eg if I know that the good bot falls for the bait & tackle I could program that in to take advantage of its weakness), and I imagine that would be ok as it's not deceiving, as it's simply playing differently against a certain opponent.
|
|
IP Logged |
|
|
|
99of9
Forum Guru
Gnobby's creator (player #314)
 
Gender:
Posts: 1413
|
|
Re: Easter eggs and gap between humans and bots
« Reply #8 on: Apr 20th, 2011, 9:27pm » |
Quote Modify
|
on Apr 20th, 2011, 3:54pm, omar wrote:
3. I consider it OK for a bot to play different moves from the exact same game state.
|
|
If you had prohibited this, I would have called the Arimaa challege a fraud! Humans could simply follow a single screening win against the bot (much like Gnobot tried to follow previous games against other bots). I'm glad that rbarreira has put in explicit randomness - that's what I was hoping to force other bot developers to do! (partly because it makes bot games more interesting for humans to play)
|
|
IP Logged |
|
|
|
tize
Forum Guru
Arimaa player #3121
Gender:
Posts: 118
|
|
Re: Easter eggs and gap between humans and bots
« Reply #9 on: Apr 23rd, 2011, 5:24pm » |
Quote Modify
|
After seeing the game between bomb and gnobot in the 2009 wcc I think that all developers that didn't already have randomness, at least added a note in the todo list about randomness. I know I did.
About marwins bait'n'tackle openings, he pick that at random but the probability depends on the opponent. Humans have a low probability and unknown bots and bots known to fall for it, they have a higher. And the bots that doesn't fall for the bait have zero probability. Apart from being fun to implement I was hoping for something like what 99of9 hoped for, that is forcing other developers to implement protection for it.
|
|
IP Logged |
|
|
|
|
Home
Help
Search
Members
Login
Register
Reply
Notify of replies
Send Topic
Print