Welcome, Guest. Please Login or Register.
Mar 29th, 2024, 12:37am

Home Home Help Help Search Search Members Members Login Login Register Register
Arimaa Forum « 2012 Computer Championship »


   Arimaa Forum
   Arimaa
   Events
(Moderator: supersamu)
   2012 Computer Championship
« Previous topic | Next topic »
Pages: 1 2 3 4  ...  7 Reply Reply Notify of replies Notify of replies Send Topic Send Topic Print Print
   Author  Topic: 2012 Computer Championship  (Read 8088 times)
omar
Forum Guru
*****



Arimaa player #2

   


Gender: male
Posts: 1003
Re: 2012 Computer Championship
« Reply #15 on: Nov 28th, 2011, 12:03am »
Quote Quote Modify Modify

We really should stick with the published rules for this year. But I know it can be a bit tedious to have your bot play all the qualifying games. Since there are only 4 bots registered this year, I am open to using previous years performance for the rankings if a majority of the registered bot developers want to use that to avoid the qualifying games. The format of the tournament will still be FTE.
IP Logged
rbarreira
Forum Guru
*****



Arimaa player #1621

   


Gender: male
Posts: 605
Re: 2012 Computer Championship
« Reply #16 on: Nov 28th, 2011, 2:26am »
Quote Quote Modify Modify

Thanks for the clarification Omar. I prefer that the seeding be done via the qualifying games.
 
I would be fine with Arimaazilla and/or Aamira2006P2 being eliminated from the list of benchmark bots though. They are too weak to contribute much (or likely, anything) to the ranking.
« Last Edit: Nov 28th, 2011, 2:33am by rbarreira » IP Logged
Nombril
Forum Guru
*****



Arimaa player #4509

   
Email

Gender: male
Posts: 292
Re: 2012 Computer Championship
« Reply #17 on: Nov 28th, 2011, 6:07pm »
Quote Quote Modify Modify

Not that I have a bot to enter... so maybe my opinion doesn't count... but it seems that seeding based on last year's performance will penalize the developers that have made the most improvements in their bots.
 
I know I had similar misgivings about having ratings play any significant role in the human tournament.  It seems the point of a tournament is to establish the best player at the time of the tournament, not crown the player that has already accumulated a high rating.
IP Logged

rbarreira
Forum Guru
*****



Arimaa player #1621

   


Gender: male
Posts: 605
Re: 2012 Computer Championship
« Reply #18 on: Nov 29th, 2011, 4:50am »
Quote Quote Modify Modify

I was looking at the rules page and saw the following:
 
Quote:
This tournament format is designed only to clearly recognize a first place winner.

 
But it's also important to recognize the second place, as that bot goes to the Challenge screening together with the winner? So I guess a tie break might be needed if there's a tie for 2nd place.
« Last Edit: Nov 29th, 2011, 4:50am by rbarreira » IP Logged
Fritzlein
Forum Guru
*****



Arimaa player #706

   
Email

Gender: male
Posts: 5928
Re: 2012 Computer Championship
« Reply #19 on: Nov 29th, 2011, 7:51am »
Quote Quote Modify Modify

It would be silly indeed to have the second bot for the qualifying phase be determined by a tiebreaker, or to have a tie for 2nd-3rd not be broken!
IP Logged

aaaa
Forum Guru
*****



Arimaa player #958

   


Posts: 768
Re: 2012 Computer Championship
« Reply #20 on: Nov 29th, 2011, 7:51am »
Quote Quote Modify Modify

on Nov 29th, 2011, 4:50am, rbarreira wrote:
But it's also important to recognize the second place, as that bot goes to the Challenge screening together with the winner? So I guess a tie break might be needed if there's a tie for 2nd place.

This is a good point, which I had forgotten about. Here is my earlier proposal to handle that:
 
on Mar 7th, 2011, 11:11am, aaaa wrote:
In case one just has to break a tie, which in our case is currently necessary to determine which bot will join the champion in the screening period, one useful technique could be to manually extend the tournament as follows:
Continue to make use of the file containing all the games of the tournament, starting by explicitly eliminating everyone not in the running, i.e. the winner and, here, all those not amongst the losers with the most wins. Because that would normally eliminate everyone, the program will revive all not-explicitly-removed players, giving them each one more life (provided the number of lives supplied as argument is no more than the number used during the original course of the tournament plus one), and schedule pairings, still taking into account the earlier games of the tournament with respect to byes, pairings and performance ratings. If further rounds are necessary, then just continue to update the history file as normal.
I would think that having the games of a main tournament affect the scheduling of a mini-tournament for runner-up in this fashion should actually be a desirable thing.
IP Logged
tize
Forum Guru
*****



Arimaa player #3121

   


Gender: male
Posts: 118
Re: 2012 Computer Championship
« Reply #21 on: Nov 29th, 2011, 3:39pm »
Quote Quote Modify Modify

If we would like to have the qualification games for seeding but decrease the "burden" on the developers, we could allow the developer to accept last years score against a qualification bot by not playing it at all. If at least one game is played against the bot then the developer has passed that offer.
 
It would at least lower the number of games needed for the returning developer.
IP Logged
aaaa
Forum Guru
*****



Arimaa player #958

   


Posts: 768
Re: 2012 Computer Championship
« Reply #22 on: Nov 30th, 2011, 8:08am »
Quote Quote Modify Modify

If this is going to be a transitional event cycle anyway, we might as well take advantage of the comparably little number of entrants and use this championship to set the groundwork for the desired future tournament format for which a consensus has already existed for quite some time now: round robin with losses carried over to a floating triple elimination phase.
 
As people have already remarked several times before, bot games are already not very attractive to witness live with their long time controls, so there should be no need to make any compromises towards any prospective spectators; the games should be scheduled and run automatically as much as possible, maximizing their possible number in any time period. The two important problems to look out for here are making sure that no leftover processes are stealing resources during games and that time lost during the communication of moves are not taken off the clock.
« Last Edit: Nov 30th, 2011, 8:09am by aaaa » IP Logged
rbarreira
Forum Guru
*****



Arimaa player #1621

   


Gender: male
Posts: 605
Re: 2012 Computer Championship
« Reply #23 on: Nov 30th, 2011, 9:01am »
Quote Quote Modify Modify

on Nov 30th, 2011, 8:08am, aaaa wrote:
The two important problems to look out for here are making sure that no leftover processes are stealing resources during games and that time lost during the communication of moves are not taken off the clock.

 
Two related questions for Omar:
 
1- Would it be hard to have a script kill off any remaining bot processes before each scheduled game? As the processes are running on the bot's own accounts, it should be enough to kill all processes on bot accounts.
 
2- If this were done, would it fix the biggest issue you have with the round robin + FTE format? As jdb pointed out in the chat, the good thing about the round-robin phase is that even if there are problems with any particular game, the other scheduled games can still go on. This is in contrast to the current format which requires immediate manual intervention at each failed game (the result is soon needed for the next round's pairings).
 
Due to that, I wonder if aaaa's suggestion could be a win-win situation both for Omar and the bot developers. Omar would not need to be on-call for each tournament game, and I know several developers would prefer a tournament format with more games (in order to make the result more representative).
« Last Edit: Nov 30th, 2011, 9:10am by rbarreira » IP Logged
omar
Forum Guru
*****



Arimaa player #2

   


Gender: male
Posts: 1003
Re: 2012 Computer Championship
« Reply #24 on: Dec 1st, 2011, 10:04pm »
Quote Quote Modify Modify

on Nov 29th, 2011, 4:50am, rbarreira wrote:
I was looking at the rules page and saw the following:
 
But it's also important to recognize the second place, as that bot goes to the Challenge screening together with the winner? So I guess a tie break might be needed if there's a tie for 2nd place.

 
Yes, you are right. We will have to resolve second place if there is a tie.  
 
Looks like aaaa, had already considered how to resolve this, so we can go with that suggestion. I'll update the page to include it.
IP Logged
omar
Forum Guru
*****



Arimaa player #2

   


Gender: male
Posts: 1003
Re: 2012 Computer Championship
« Reply #25 on: Dec 1st, 2011, 10:20pm »
Quote Quote Modify Modify

on Nov 30th, 2011, 9:01am, rbarreira wrote:

 
Two related questions for Omar:
 
1- Would it be hard to have a script kill off any remaining bot processes before each scheduled game? As the processes are running on the bot's own accounts, it should be enough to kill all processes on bot accounts.
 
2- If this were done, would it fix the biggest issue you have with the round robin + FTE format? As jdb pointed out in the chat, the good thing about the round-robin phase is that even if there are problems with any particular game, the other scheduled games can still go on. This is in contrast to the current format which requires immediate manual intervention at each failed game (the result is soon needed for the next round's pairings).
 
Due to that, I wonder if aaaa's suggestion could be a win-win situation both for Omar and the bot developers. Omar would not need to be on-call for each tournament game, and I know several developers would prefer a tournament format with more games (in order to make the result more representative).

 
Yes, a script could be setup to kill the processes before a game starts.
 
But, I would still prefer not to use a round robin. It's not too bad with 4 bots, but if we had 8 it would be way too many games. However, I would like to encourage the bot developers to make use of the tournament management tool to organize their own tournaments outside of the WCC.
 
IP Logged
omar
Forum Guru
*****



Arimaa player #2

   


Gender: male
Posts: 1003
Re: 2012 Computer Championship
« Reply #26 on: Dec 1st, 2011, 10:25pm »
Quote Quote Modify Modify

on Nov 28th, 2011, 6:07pm, Nombril wrote:
Not that I have a bot to enter... so maybe my opinion doesn't count... but it seems that seeding based on last year's performance will penalize the developers that have made the most improvements in their bots.
 
I know I had similar misgivings about having ratings play any significant role in the human tournament.  It seems the point of a tournament is to establish the best player at the time of the tournament, not crown the player that has already accumulated a high rating.

 
True. But sometimes I think we worry too much about seeding. The difference between a randomly seeded FTE and a perfectly seeded FTE is not that much in terms of the formats chance of selecting the best player. I had posted the results of such simulations once, but can't seem to find them right now.
IP Logged
rbarreira
Forum Guru
*****



Arimaa player #1621

   


Gender: male
Posts: 605
Re: 2012 Computer Championship
« Reply #27 on: Dec 3rd, 2011, 10:38am »
Quote Quote Modify Modify

on Dec 1st, 2011, 10:20pm, omar wrote:

 
But, I would still prefer not to use a round robin. It's not too bad with 4 bots, but if we had 8 it would be way too many games.

 
If the length of the tournament is the biggest issue (and I understand why it would be) here's an idea:
 
What about using Fast (30s per move) games for the round-robin phase, and move on to the regular time control of 2 minutes per move only during the FTE phase which has much fewer games (especially with losses in the round-robin carried forward)?
 
30s per move games would be four times faster than 2 minutes per move, so it would be possible to pack 12 games into a single day of the tournament. Even with 8 bots, the double round-robin phase could be played in less than 5 days.
 
Of course this assumes that bot developers are fine with having their bot playing Fast games in the round-robin phase. I for one would not mind it... In the human championship (or at least past ones) there were already different time controls for different phases of the tournament.
« Last Edit: Dec 3rd, 2011, 10:39am by rbarreira » IP Logged
aaaa
Forum Guru
*****



Arimaa player #958

   


Posts: 768
Re: 2012 Computer Championship
« Reply #28 on: Dec 5th, 2011, 5:27pm »
Quote Quote Modify Modify

I've changed my mind. Perhaps a uniform tournament structure is better after all, just like with the human championship. Quadruple elimination scales better than the hybrid format; the former will have much more "relevant" games, whereas the latter could waste lots of them on already-doomed weaker bots. If the number of games ever were to become an issue, then one could just easily decree that the number of lives should be a function of the number of participants, just like how it was with the number of rounds in the Swiss Open Classic.
IP Logged
rbarreira
Forum Guru
*****



Arimaa player #1621

   


Gender: male
Posts: 605
Re: 2012 Computer Championship
« Reply #29 on: Dec 6th, 2011, 11:13am »
Quote Quote Modify Modify

on Dec 5th, 2011, 5:27pm, aaaa wrote:
I've changed my mind. Perhaps a uniform tournament structure is better after all, just like with the human championship. Quadruple elimination scales better than the hybrid format; the former will have much more "relevant" games, whereas the latter could waste lots of them on already-doomed weaker bots. If the number of games ever were to become an issue, then one could just easily decree that the number of lives should be a function of the number of participants, just like how it was with the number of rounds in the Swiss Open Classic.

 
I would prefer quadruple elimination (FQE) over FTE too. The top bots seem to be quite close which makes the current format not very discriminating.
 
To illustrate that point, I fed the 2010 WCC and 2011 WCC games (separately) into BayesElo and it gave a probability of superiority of the winner over the 2nd place of just 67% and 58% respectively. In other words this tournament format is not much better than a coin toss in terms of telling which is better between the two top bots (not to mention a case where three top bots are close enough, where one of them has to be out of the challenge screening).
« Last Edit: Dec 6th, 2011, 11:22am by rbarreira » IP Logged
Pages: 1 2 3 4  ...  7 Reply Reply Notify of replies Notify of replies Send Topic Send Topic Print Print

« Previous topic | Next topic »

Arimaa Forum » Powered by YaBB 1 Gold - SP 1.3.1!
YaBB © 2000-2003. All Rights Reserved.