|
||||||||||
|
Title: 2007 Computer Championship and Challenge Post by omar on Oct 7th, 2006, 2:26am I've finished editing the pages for the 2007 Computer Championship and Challenge match (I think). http://arimaa.com/arimaa/wcc/2007/ http://arimaa.com/arimaa/challenge/2007/ Please have a look and let me know if you have any feedback. Thanks. |
||||||||||
|
Title: Re: 2007 Computer Championship and Challenge Post by unic on Oct 7th, 2006, 3:25pm Fairy won't be there (unless somebody else enters it...) - I'm doing programming for fun, not for research, and modifying Fairy to follow these conditions would not be fun for me: Quote:
If somebody else wants to modify Fairy so that it fulfills the conditions above, and enter it, feel free to do so. Source is at the download page (I have a newer experimental version... but that one is still weaker than the old one.) |
||||||||||
|
Title: Re: 2007 Computer Championship and Challenge Post by omar on Oct 7th, 2006, 4:14pm The first and second conditions are fairly simple: * Provide a README file for how to run the bot * Statically compile using the -static option (if program is in C or C++); not needed for Java programs. The third one is automatically there if you used the C sample bot code. Otherwise, just add an option to have the bot stop after a specified search depth or number of position evals or something else so that its play streangth can be fixed using such parameters. The last two are not needed if your bot does not use any random number generator. If it does then it may produce a different move if invoked with the same position a second time. The reason for these conditions was so that it could be proved that the a particular move was produced by the bot by giving it the same game state and same seed for the random number generator (assuming the bot logged to a file what the seed value was) and having it produce the same move again. However, thinking about this again, I just realized that the small variations in the server load could also cause a slight fluctuations in the number of positions a bot evalutes from one run to another even the clock time for the move is the same and thus cause it to produce different moves even if all the initial conditions were the same. Thus, I will eliminate these last two conditions. Hopefully the other conditions aren't too much of a burden for the developers. |
||||||||||
|
Title: Re: 2007 Computer Championship and Challenge Post by Fritzlein on Oct 8th, 2006, 11:11am It would be a shame if Fairy didn't compete in the computer championship, especially if (as expected) there are fewer than eight entrants. If unic feels it is too great a burden to code in some mechanism to limit the search depth of Fairy, thus not qualifying, I hope that someone else will take the earlier, published version of Fairy, add the depth-limit feature, and enter that version of Fairy just for benchmarking. Even if the developer getting Fairy ready for entry was entering his own bot (99of9, jdb, doublep, etc.) it seems totally in the spirit of the rule "one bot per developer" to have one version of Fairy playing, one way or another. |
||||||||||
|
Title: Re: 2007 Computer Championship and Challenge Post by Fritzlein on Oct 8th, 2006, 11:19pm on 10/07/06 at 02:26:53, omar wrote:
I guess you mean http://arimaa.com/arimaa/wcc/2007/ The one issue I have with the format is what to do in case of a tie for second place based on the number of wins. The rules now say a tie will be broken by pre-tournament rating, but why not have an elimination playoff between anyone involved in a tie for second? If there are N players tied for second, it only takes N-1 more games to have a playoff. (Usually it will be two players tied for second, if there is a tie at all, and thus require only one playoff game.) Quote:
In the first paragraph you say that the prize is currently $15,000, but that leaves out $1,500 pledged from PMertens and $600 from me, for a total of $17,100. At least a couple of community members are taking your side of the bet! I would also try to make this wording more clear: "Any human player that wants to play against the programs must play both programs equally in terms of number of times and color. A human player must not play more than two times against either program and no more than one time with the same color. Thus, if a human player plays program A as gold, the player must then play program B as gold." Rather than saying players "must" do this or that, is it possible for you to technically enforce the rule? Can you set the bots up to only accept invitations from humans they haven't yet played with that color? If so, then I would say, "A human player who wishes to participate in the qualifying process will be permitted to play each of the two contenders once with each color, for a total of four games. A game does not count for the standings until the same human plays the other contender with the same color." Perhaps the ideal mechanism is to have a link that says "Participate in the Challenge qualifying process", such that clicking on that link will pair the human appropriately with a contender and color to play. I like that the defenders of the Challenge are barred from these qualifying games. I would also bar the developers of the two contending bots from playing in these qualifying games, because they have a strong incentive to try to skew the standings by losing to their own bot. I think that will still leave plenty of possible opponents, given the current size of the Arimaa community. For good measure, I would include a rule that any games which appear to have been intentionally lost in order to distort the standings may be excluded at your discretion. I see that you have listed the time control for all the qualifying and challenge games as 2/2/100/10/8 . I can understand having a long time control in order to give humans the greatest possible advantage, but I don't think humans need the slow time control this year. Why not have the games be at 90 seconds per move? This would make it easier for the human defenders (less time commitment) without significantly affecting winning chances, and would make it more interesting for spectators (less boring wait for the computer to move). Also the faster time control would probably get more people to play the two bots during qualifying, and thus give us a better idea of which bot is really stronger against humans. You could always change the time control back to two minutes per move in future years if it seems the humans might need it, and if Arimaa community has grown to the point that there is a surplus of people wanting to play in the qualifying games. I understand that a slow time control is appropriate to let humans give their best defense of their own intelligence before a program ultimately wins the Challenge. In the mean time, however, speeding up the games to 90 seconds per move is good for spectators, and good for participants, and it makes the dominance of humans all the more clear. |
||||||||||
|
Title: Re: 2007 Computer Championship and Challenge Post by omar on Oct 9th, 2006, 8:36pm on 10/08/06 at 23:19:07, Fritzlein wrote:
Yes, that's what I ment. Thanks. I've fixed it. |
||||||||||
|
Title: Re: 2007 Computer Championship and Challenge Post by omar on Oct 10th, 2006, 8:45am on 10/08/06 at 23:19:07, Fritzlein wrote:
But in the tournament rules I would need to specify it for all cases (like 3 or 4 players tied for second) and that gets kind of messy so I thought lets just use the pre-tournament rating. But thinking about it again, I guess we could just say: in case of ties for second place a single elemination fold tournament will be used to determine second place. Byes will be given to the player with the highest pre-tournament rating and the lower rated player within a pairing will chose the color assignment. I've changed it to this now. Quote:
I was under the impression that those pledges were only good for one year. If that's not the case please let me know and I will add them to the total. Quote:
Yes, that's exactly what I was planning to do. I'll consider changing the wording. Quote:
Good ideas. I've added them to the challenge page. But instead of me we should let the match director decide if the game appears to be thrown. Quote:
We can't change the time controls later. I am working towards fixing them based on experience now, but once they are fixed we can't change them or anything else about the match. |
||||||||||
|
Title: Re: 2007 Computer Championship and Challenge Post by Fritzlein on Oct 10th, 2006, 4:56pm on 10/10/06 at 08:45:34, omar wrote:
Oh, well, if you are going to fix the time control for all the remaining years of the Challenge, then I would go with two minutes per move. Even slower than that would help the humans more, of course, but two minutes is extremely slow already. It's what I think of as the slowest reasonable time control. However, before you set the rules in stone, do you want to at least experiment with 2/12/75/0/8/5 ? So far the main argument against allowing reserve to accumulate beyond the max time per move is that it is experimental, but if we never even experiment with it, how will we know? By the way, one thing that is even more experimental is the method of having the top two Computer Championship bots contend for the right to be Challenger. I'd be extremely leery of setting that in stone unless we try it for at least two years. It seems much harder to test the selection method than to test a time control, because the time control would be tested nine times in a single match, whereas the selection process is only tested once per match. Quote:
Ah, I had just imagined the second-place playoff being a continuation of the main tournament. Everyone tied for second place gets an extra life, and they play on as if they were the only ones left in the tournament. This is the same as the single-elimination you are proposing, except that it retains the memory of repeat pairings and color assignments from games already played. The difference is small, but since the two formats are conceptually equally easy to state, why not go with the fairer one? Quote:
For last year I made a pledge of $500, good for only one year. Then after nobody won that, I made a pledge of $600, good for only one year (i.e. this year). After this challenge match I expect to make another pledge, perhaps for a lesser dollar amount, but good until 2020, to simplify the bookkeeping. Here is PMertens' pledge: on 10/13/05 at 01:31:25, PMertens wrote:
I don't think anyone else has pledged, so we get $15000 + $1500 + $600 = $17100. |
||||||||||
|
Title: Re: 2007 Computer Championship and Challenge Post by omar on Oct 11th, 2006, 8:27am on 10/10/06 at 16:56:13, Fritzlein wrote:
How about the possibility of allowing the human player to select the time control, but it must have an average time-per-move between 1 to 3 minutes. This way I don't have to fix an exact time control. But there are a lot of pros and cons associated with this. We might need to start a new topic to discuss it. However, for this year we will stick with the 2/2/100/10/8 time control, but discuss this for possible use in future years. Quote:
Yes, if we find problems with this, we will have to amend it or use something different. Quote:
Yes, this would probably be more fair, but the problem is I don't have a program yet that can do the pairing in this situation. So for now we'll just use single elimination. Besides I think the chances of having more than 2 players tie for second place when starting with 16 players is fairly low that we don't really need to worry about this too much. Anyone want to take a crack at figuring out the probability. Quote:
Thanks; I've updated the challenge page. |
||||||||||
|
Title: Re: 2007 Computer Championship and Challenge Post by Fritzlein on Oct 11th, 2006, 1:15pm on 10/11/06 at 08:27:21, omar wrote:
On further inspection this is simple enough you don't need to worry about having a computer program to pair it. The number of wins a player has at the end of a tournament is equal to the round he was eliminated minus the number of losses minus the number of byes received. For all eliminated players the number of losses is the same, so we can simply compare round eliminated minus number of byes to determine second place. The player who is eliminated last by definition has survived one round longer than all other eliminated players. Therefore this player will win second outright unless he has recieved one more bye than a player eliminated the previous round. At most two players can have been eliminated the previous round, so there can at most be a three-way tie for second place. A three-way tie for second seems rather unlikely, since it requires * The tournament must have exactly four players left at some point. Sometimes this won't happen because eliminations will move the tournament directly from five or six players down to three. * In the round of four players, exactly two players must be eliminated. Sometimes this won't happen because one of the two losers in that round has an extra life left to play on * The two players eliminated must both have one bye fewer than a remaining player on the verge of elimination * That remaining player with an extra bye on the verge of elimination must lose the next round. I'm guessing less than 5% chance of a three-way tie for second place. A two-way tie for second place however, seems much more likely. In both the World Championship and the Computer Championship last year, there was a clear second place winner based on number of wins, but there could have been a tie for second place had one game gone differently, and another game gone as expected. Maybe there's about a 20% chance of a two-way tie for second place. A two-way tie for second place requires only a one-game playoff, and color assignment can be done by hand based on colors in the tournament so far. A three-way tie for second place could be resolved by having the last-eliminated played take on the winner of the two others. (Note that pairing the two players eliminated in the second-last round is unlikely to create worse repeat-pairing scenario than another pairing. Those two didn't play each other on the round they both were eliminated, and one of them played the later-eliminated player.) That's two games, and again the color assignment could be done by hand for each game. In short, the tie-break algorithm could be sufficiently straightforward that computer assistance is not required. |
||||||||||
|
Title: Re: 2007 Computer Championship and Challenge Post by unic on Oct 12th, 2006, 1:32pm Regarding time controls: I think shortening it is a bad idea. I also think limiting time per move is a bad idea - compare most other mind sports that are played seriously (chess, shogi, go); knowing when to spend half an hour (or more - in professional shogi games, I've seen times up towards a couple of hours on one single crucial move) on a move and when to move quickly - in other words, deciding how to use one's time - is an important skill. If a player plays quickly for a while and thus builds up a significant reserve of time, I think the player should be able to use it - limiting reserve time to a maximum, or putting a maximum time per move, serves to remove a skill element from the game. Also, anything less than 2-3 minutes per move average time strikes me as a bit on the fast side... especially as Arimaa doesn't have much of an opening phase where the players can move quickly, saving up time for later (compare with chess...) I'm dubious about letting the top two programs contend for the right to challenge the human - while there are advantages (the best program against other programs might not be the best against humans), the method given for selecting the challenging program seems... inexact and unfair to me. (Oh, and about Fairy, if I manage to actually get a version that's significantly stronger than the current one, I might reconsider entering it... but for now, I can't see the point to spend effort on that while Fairy is still so much weaker than the top programs - but if somebody else wants to modify/enter it, feel free to do so.) |
||||||||||
|
Title: Re: 2007 Computer Championship and Challenge Post by Fritzlein on Oct 12th, 2006, 4:39pm Thanks for joining the discussion, unic. It's good to have someone take up the banner for long thinking time and no restrictions on its allocation. MrBrain used to argue for this, but he hasn't been around for a while. I agree that time management is indeed a skill, and any limitations on the time management of the players removes some of that skill. I know almost all chess players like to have full control over their time. What do you think about the desires of spectators? Do you think the comfort of the players should be balanced somewhat against the entertainment of the onlookers? If you look at what sponsors of chess tournaments want, they generally want the games to speed up for the sake of the spectators, because spectators generally want something to happen at least every couple of minutes. on 10/12/06 at 13:32:24, unic wrote:
Inexactness is going to be present with any method of choosing a challenger, for example an inferior bot winning the Computer Championship by an upset, so unless the problem is worse than straight triple-elimination among bots, the inexactness doesn't bother me so much. But unfairness bothers me much more. Can you expand more on why you think it is unfair? Quote:
Thank you for your blessing on getting the public version of Fairy entered. It will be a great addition to the tournament even if it doesn't win. |
||||||||||
|
Title: Re: 2007 Computer Championship and Challenge Post by omar on Oct 14th, 2006, 8:38am on 10/11/06 at 13:15:30, Fritzlein wrote:
Thanks for analyzing this Karl. I have updated the WCC and WC rules page to use this for determining second place. |
||||||||||
|
Title: Re: 2007 Computer Championship and Challenge Post by unic on Oct 14th, 2006, 9:57am on 10/12/06 at 16:39:10, Fritzlein wrote:
... that's where I am an atypical spectator. To me, the focus should be on the contest of skills - finding out which of the players is the better one - not on entertaining the spectators. And I (as a spectator) get more enjoyment out of watching a game the higher its level is - which means that for chess, e.g., I find 25-minute games boring to watch, while I sat for hours following the classical games of the recent world championship match. In my opinion, the spectators shouldn't be catered to. The competition to find out who is best should be in center; not the spectators. If they want to watch, then that's good... if they don't want to watch, well, I don't see it as a problem if there are no spectators. (Incidentally, I follow Formula-1 racing, and am very annoyed with the same tendency there - changing the sport so that its focus is on providing "entertainment", not on finding out which team/driver is the most skillful. In F1, the tendency is even stronger than in chess; presumably because of the larger amount of money involved. On a personal level, the changes made in order to make it more entertaining has partially caused me to lose my interest - I watch much fewer of the races now than I used to.) So... I guess the question is - what is the focus of the competition? To find out the best player, or to provide entertainment? For me, it is to find out the best player - if I want entertainment, there are many other ways I can get that. (And, on a related note, I don't see why the computer championship can't use a better method for selecting the champion - multiple round-robin should be more exact, and the bots shouldn't mind having to play many games - they're doing it automatically, after all. Again, triple-elimination seems to be more "spectator-friendly" - few games are played, so easy for humans to follow, and plenty of "critical" games, where a program must win or be eliminated. Multiple round-robin, on the other hand, has lots of games, and not many critical... but precisely this means any single game has less impact, and thus the selection of the champion is more exact.) Quote:
Inexactness is going to be present with any method; however, the amount of it depends on which method is used. Again, what is the aim of the contest? To find a good approximation with the fewest number of games (in which case, triple-elimination works well), or to find a better approximation even if it takes a huge number of games (which would be my preference). The unfairness in my eyes is simply the lack of control around the structure. Unfairness might be too strong a word... I see it as a large amount of inexactness beyond the influence of the competing bots. |
||||||||||
|
Title: Re: 2007 Computer Championship and Challenge Post by Fritzlein on Oct 16th, 2006, 11:36am Thanks for explaining, unic. I do see a point about how catering to spectators can make a game less interesting. As a baseball fan, I hate the way parks have moved the fences in. They think more home runs means more excitement, and that that will keep the fans happy. I think the home run is the most boring play in basball, apart from a strikeout or a walk. For a walk, a strikeout, or a home run, there are seven players standing out in the field (plus anyone on base) doing nothing. Even a pop-up or a grounder is more exciting because fielders and baserunners have to make decisions, and have the possibility of errors. What would put the excitement into baseball is to move the fences back 200 feet. That would keep every ball in play, and make home runs (inside-the-park) truly exciting. As an Arimaa spectator, I would hate for the match conditions to take strategy out of the game. I would definitely not want Challenge games played at blitz speeds. I would not want games to be primarily decided by who hung a piece, rather than by who had the better plan. Also, as a spectator, I like to have time to see what the plans are and understand them a bit between moves. Maybe chess grandmasters could play a blunder-free game at ten seconds per move, containing deep strategies and profound moves, but I sure couldn't appreciate it at ten seconds per move. In short, I do appreciate the argument for a slower time control. This is, however, quite separate from the issue of giving players total control of the management of their time. I don't like the way chess players blitz through their opening moves to save up time for the midgame, because that doesn't let me savor their options in the opening. I don't like how often players have to blitz to get to time control because they thought too long in the midgame. It is optimal for my enjoyment as a spectator if the moves are spaced evenly one to three minutes apart: enough time to analyze and enjoy, but not enough for boredom. That's why, even at a slow time control, I favor the other parameters to encourage play at a steady pace. on 10/14/06 at 09:57:58, unic wrote:
Here I totally disagree. It does matter if people like to watch the game, because it indirectly has a huge impact on my opportunity to enjoy the game. An absence of fans of the game means no money and fewer opponents to play. I know you understand this problem perfectly well yourself: you said in another post that you know of "better" games where you can't find opponents, thus you are here and not there. Maybe if those "superior" games had a greater focus on entertainment, they would be more popular than Arimaa. Quote:
If you focus 100% on entertainment, then I agree with you, the game will lose its focus on skill. It might be entertaining to have secret landmines planted it the board, or to roll dice to determine how many steps you get to take, or other such silliness. But things that take skill out of the game ultimately take the fun out of the game too, and we would be left more bored and less entertained than before. On the other hand, if you focus 100% on skill, then you can squeeze the fun out in other ways. Maybe there would be more skill on a 12x12 board than on an 8x8 board, but the games would take days to play, and that wouldn't be fun. Maybe there would be more skill if there were less capturing, but that might again slow the game down to where it takes 200 moves to win, and thus is supremely boring. You just can't leave the fun out of the equation, or nobody will play your game that is an awesome test of skill. Taking a principled stand that skill is the only thing that matters would be just bad for Arimaa as saying that entertainment is the only thing that matters. Neither is ultimately fulfilling. In my mind we're trying for a balance. Yes, limiting each move to a 5-minute maximum squeezes a tiny bit of skill out of the game. On the other hand (for the vast majority of us) allowing a single move to take half an hour squeezes a whole lot of fun out of the game. A five-minute per-move limit is good for Arimaa on balance. Quote:
|
||||||||||
|
Title: Re: 2007 Computer Championship and Challenge Post by omar on Oct 20th, 2006, 8:48am on 10/14/06 at 09:57:58, unic wrote:
Ideally I agree with you 100% on this. But practically, not catering to the spectators gives you a 100% chance of killing any game, good or bad. Any game that involves skill depends on the spectators for it's long term survival. No spectators leads to no sponsors or ticket income which means less prize money which results in fewer talented players wanting to devote their time to the game. Of course without players a game is basically dead. Also the other thing to realize is that at every speed there is some player or players that excell at that speed. Yes, the quality of the moves decreases with faster speed along with the chance of the same player winning the tournament again, but that's something we have to accept to a certain degree if we want to make the game interesting for spectators. The other thing is that as the difference between two players gets smaller it will take longer and longer to distinguish which player is truely better. At some point the amount of effort needed to make that distinction out weighs the reward. Even with computers, the amount spend on dedicating the hardware has a cost. My general approach is try to be as ideal as possible while catering to the conflicting requirements of different groups, because in the end they all matter. |
||||||||||
|
Title: Re: 2007 Computer Championship and Challenge Post by omar on Oct 20th, 2006, 8:57am on 10/16/06 at 11:36:01, Fritzlein wrote:
As I had mentioned in an earlier discussion I actually think rolling a dice for the number of steps would be in interesting variation to Arimaa. In fact some of the early version did use a dice. Adding a little luck to the game does not completely kill skill, but does give more players a higher chance of winning and thus allows a game to prosper on just players alone without the need for spectators. Poker for example; and the lottery is an extream example. |
||||||||||
|
Title: Re: 2007 Computer Championship and Challenge Post by Fritzlein on Oct 20th, 2006, 9:41am on 10/20/06 at 08:57:21, omar wrote:
Luck games usually thrive because people like to gamble with money. The lottery thrives with no element of skill whatsoever because people like to dream of winning millions. If there were no money involved, nobody would play the lottery just for fun. Poker is a different story. There are many levels of skill in Texas Hold'em. However, it is just as difficult to create strategic depth in a luck game as it is in a game with no luck. Over the years there have been hundreds and hundreds of poker variations that have died off because there was no depth. Everyone could master all the skill of a certain variation in a short time, and then it became totally luck, i.e. you might as well flip a coin as play that variation of poker. Five-card stud is one of the good variants of poker, and it used to be very common way to play for high stakes. But eventually the depth of five-card stud was exhausted, everybody learned how to play it well, and that universal knowledge of strategy killed it off. Hardly anyone plays five-card stud any more. Texas Hold'em, in contrast, is not well understood even at the highest level of play. Not long ago I saw Phil Hellmuth ranting at two guys who beat him in the finals of a tournament because (in Phil's opinion) they played poorly and got lucky. In other words, the world's best players of Texas Hold'em disagree about proper strategy just as much as top Arimaa players disagree about proper strategy. While it would be theoretically possible to introduce luck into Arimaa in a way that preserves the depth and skill, I highly doubt it could be done in practice. Most likely you will kill off many layers of depth and replace them with a single, simpler idea. Adding dice to Arimaa would (in my estimation) be 99% likely to create a game that has superficial interest but little depth, just like 99% of variations of poker. In other words, you would probably make a lottery out of the game. As soon as everyone masters the skill of dice-Arimaa, you are left with essentially a complicated way of flipping a coin. The present rules to Arimaa are something very rare, indeed almost miraculous. I don't think I could design a game as good as Arimaa if I spent my lifetime trying. Please, don't mess with it unless you absolutely have to (i.e. unless some flaw is killing the game anyway). I must believe that changing something this good is more likely to make it worse than better. |
||||||||||
|
Arimaa Forum » Powered by YaBB 1 Gold - SP 1.3.1! YaBB © 2000-2003. All Rights Reserved. |