Arimaa Forum - Print Page


    
      
        Arimaa Forum
        (http://arimaa.com/arimaa/forum/cgi/YaBB.cgi)
      

        Arimaa >> Bot Development >> Goal Test Positions
        
(Message started by: rabbits on Nov 5^th, 2011, 12:43am)

Title: Goal Test Positions
Post by rabbits on Nov 5^th, 2011, 12:43am

When I was working on the static goal search for my bot a year and a half ago, I hand crafted a bunch of test positions. While skimming the chat archive the other day, I saw some talk about test positions, so I figured I might as well share mine.

Goal Test Positions (http://www.math.utk.edu/~gclark/arimaa/goal-test-positions.tar.gz)

Let me know if anyone actually tests their bot with these positions. Also, if there are any mistakes, post about those too please!

Title: Re: Goal Test Positions
Post by rbarreira on Nov 5^th, 2011, 4:22am

Thanks for posting those! I definitely want to try them out but I'll need some automatic script to run my bot with them (and check the answer), as they're so many.

Why do some positions have "cangoal true" and some others have "cangoal [1..4]"? Does it simply mean you didn't bother filling in how many steps the goal consists of?

Title: Re: Goal Test Positions
Post by rabbits on Nov 5^th, 2011, 12:12pm

Of course you would want to automate the testing! I had my bot setup so that if I passed it the --atp flag it would run these arimaa test positions.

I think I started out with only "cangoal true" and "cangoal false" and then added the steps needed option without changing my older tests.

That reminds me, I wrote a gedit text highlighting thing for Arimaa positions. I'll share that too :)

GTK Sourceview Arimaa color scheme / ATP language (http://www.math.utk.edu/~gclark/arimaa/arimaa-gtksourceview.tar.gz)

Here is a screenshot of what it looks like on my computer right now:
http://www.math.utk.edu/~gclark/images/highlighting.png

Title: Re: Goal Test Positions
Post by rbarreira on Nov 5^th, 2011, 12:22pm

I managed to run all the positions.

There were no false positives (i.e. the positions with "cangoal false" all worked as they should). This is good as false positives are worse than false negatives.

As I expected, there were plenty of false negatives, even more than I expected - 130 of the 303 goal positions reported no goal found. I'll be going through them and adding cases to my bot (hopefully some of those cases are redundant and I don't have to add 130 pieces of code). I'll let you know if I find any mistakes

Thanks again for the test positions!

Title: Re: Goal Test Positions
Post by rabbits on Nov 5^th, 2011, 12:36pm

You had tested your static goal search against all of the games in the Arimaa games database, right? I guess not all goaling patterns have actually occurred in real games then!

Title: Re: Goal Test Positions
Post by rbarreira on Nov 5^th, 2011, 12:49pm

My bot does not detect all goals in the Arimaa games database either. There are more than 600 games (out of around 120,000 ended in goal) where goal is not found.

That said I wouldn't be surprised if your test cases have some goals which didn't happen yet, given the high rate of not found goals I have!

Title: Re: Goal Test Positions
Post by UruramTururam on Nov 5^th, 2011, 4:24pm

on 11/05/11 at 12:12:07, rabbits wrote:

Here is a screenshot of what it looks like on my computer right now:
http://www.math.utk.edu/~gclark/images/highlighting.png

Is there a forced goal here? Wow...

Title: Re: Goal Test Positions
Post by Swynndla on Nov 5^th, 2011, 9:37pm

Thanks for sharing rabbits. :)

I installed gedit (as I don't use gnome) but it didn't work - probably because I installed gedit version 2 not 3? (as version 2 is available in debian stable but not 3) ... I copied the files to 2.0 directories instead of the (non-existent) 3.0 directories, but it still would work, eg:

Code:

(gedit:8896): GtkSourceView-WARNING **: in file /usr/share/gtksourceview-2.0/language-specs/atp.lang: style 'atp:board' not defined

etc

No big deal - I'm happy to wait until version 3 is in debian stable. :)

Title: Re: Goal Test Positions
Post by lightvector on Nov 5^th, 2011, 10:01pm

Nice set of test positions. I'll be trying sharp on them soon too. Thanks for posting them.

Rabbits: It actually is the case that being correct on the entire Arimaa game database is not sufficient to have a correct goal search. Some goaling patterns actually haven't occurred in real games! In earlier testing, a few months ago, I found that even after being correct on the entire game database, there were still a few types of goals that sharp missed.

The way I found these additional missed patterns was to not only test each position in the game database, but also a hundred or so perturbations of each position. Each perturbation took the original position and modified it by randomly sprinkling some additional rabbits and other pieces around the board, possibly overwriting existing pieces. The idea was to transform typical midgame positions into strenuous goal threat tests. This frequently happens when an enemy rabbit gets randomly placed behind your lines, or when an existing advanced rabbit gets an additional supporting piece dropped next to it, and in either case, possibly overwriting a defending piece too. Perturbing the position also worked well to detect bugs in positions where a rabbit in the original position could almost goal or barely goal, so that the perturbations would readily flip the status back and forth in varied ways.

Title: Re: Goal Test Positions
Post by rabbits on Nov 6^th, 2011, 12:17pm

UruramTururam: Haha, no there is not a forced goal in the position in the screenshot! That position is not included in the goal test positions that I shared. It's just a position I used to check that my bot was generating the correct number of legal steps and moves.

Swynndla: I actually just changed the install script to install in the GtkSourceView version 3.0 directory yesterday when I reinstalled those files, but I didn't change the files. If you look inside of the files, you will see that they are not version 3.0. As for the warnings, well, I had never noticed before, but I get those too:

Code:

(gedit:18608): GtkSourceView-WARNING **: in file /usr/share/gtksourceview-3.0/language-specs/atp.lang: style 'atp:board' not defined

(gedit:18608): GtkSourceView-WARNING **: in file /usr/share/gtksourceview-3.0/language-specs/atp.lang: style 'atp:board-labels' not defined

(gedit:18608): GtkSourceView-WARNING **: in file /usr/share/gtksourceview-3.0/language-specs/atp.lang: style 'atp:board-inside' not defined

(gedit:18608): GtkSourceView-WARNING **: in file /usr/share/gtksourceview-3.0/language-specs/atp.lang: style 'atp:gold' not defined

(gedit:18608): GtkSourceView-WARNING **: in file /usr/share/gtksourceview-3.0/language-specs/atp.lang: style 'atp:silver' not defined

Hmmmm, oops! It still works for me though...

By the way, is Debian still using Gnome2? If so, I might switch to it!

Lightvector: Cool, thanks for verifying that! I created a lot of these positions by perturbing other positions too, except that I did so manually and with the motivation to test each chunk of code as I wrote it.

Title: Re: Goal Test Positions
Post by Swynndla on Nov 6^th, 2011, 1:40pm

Oh silly me - I don't know anything about gedit ... I had to go to Preferences > Fonts & Colors and then choose Arimaa ... now it works. :)

Title: Re: Goal Test Positions
Post by rabbits on Nov 6^th, 2011, 1:43pm

Yeah, I had to make a whole "style scheme" to get the gold, silver, and brown colors I wanted...

Title: Re: Goal Test Positions
Post by Swynndla on Nov 6^th, 2011, 1:48pm

I think debian stable & testing uses gnome version 2.30 ... that's what the repo's tell me anyway? I use KDE and LXDE so I don't know much about gnome. By going to debian stable or testing, some packages wont be as up-to-date as with the latest ubuntu.

Title: Re: Goal Test Positions
Post by doublep on Nov 11^th, 2011, 2:37pm

I can share regression files from Badger if anyone wants, but they are coded in GTP (http://www.lysator.liu.se/~gunnar/gtp/) reworded for Arimaa. So a test looks like:

clear_board
setup_squares Ra7 Cb8 ra8 ec7 cb6
50010 find_one_move_win white
#? [true Cb8s-ra8e-Ra7n]

Or one found with automatic error-finding tool:

setup_board ----r---rrrMdr-rrReR----DE------c-----m-------hr---------RR-RRRR
670 find_one_move_win white
#? [true rc7n-ec6x-Md7w-Rd6n-Rd7n]

I also don't have anywhere to host files :/

Title: Re: Goal Test Positions
Post by Dolus on Nov 11^th, 2011, 3:23pm

That sounds helpful.

As far as sharing files, I expect somebody should have a good solution to this, but otherwise, we could set up a Dropbox account or something along those lines.

Title: Re: Goal Test Positions
Post by doublep on Nov 13^th, 2011, 12:31pm

on 11/11/11 at 15:23:22, Dolus wrote:

That sounds helpful.

As far as sharing files, I expect somebody should have a good solution to this, but otherwise, we could set up a Dropbox account or something along those lines.

Ah, I remembered I have a Google account. You can grab the file (https://docs.google.com/open?id=0B3GzxNlUxcpMMmVlMzc2N2EtY2ZkZS00Yzk4LWJlYWUtMTRhYjBmYWU0MTg3). It's a tarball with some Badger's tests, Python scripts and a brief GTP description.

Title: Re: Goal Test Positions
Post by JimmSlimm on Aug 17^th, 2012, 6:43pm

on 11/05/11 at 00:43:42, rabbits wrote:

Thank you so much for sharing this. I found MANY mistakes in my code when running tests on these positions.

Title: Re: Goal Test Positions
Post by Max on Dec 25^th, 2013, 4:04pm

Thank you all for your test positions.

My bot (bot_Maximus) I am developing at the moment found all goaling moves from rabbits test-positions. That is no surprise as it has only a complete search implemented at the moment. So the only problem was, it took over 50 minutes for finding/not finding the goals.

Title: Re: Goal Test Positions
Post by lightvector on Dec 27^th, 2013, 3:26pm

I'm still a big fan of randomized testing using the full games database for "seed" positions - you can get extremely thorough coverage, much more than any specifically crafted test set can give you, although there is also much more cruft (your code has to be efficient to be able to process the large number of positions in a reasonable time).

Also, one thing to note when writing the static goal detector is that rather than just producing the boolean answer or just the [0-4] number of steps to goal, if you can stand the additional complexity, it's worthwhile to make it also return the goaling move or the set of squares involved in the goal move. The reason is that when you do goal-relevance pruning (restricting move generation to just the steps that might be relevant to stopping the goal when the opponent has a goal threat), if you know the exact set of squares involved in the goal move, you can safely prune far more aggressively.

Title: Re: Goal Test Positions
Post by leon_messerschmidt on Feb 28^th, 2014, 8:02pm

on 11/05/11 at 00:43:42, rabbits wrote:

Let me know if anyone actually tests their bot with these positions. Also, if there are any mistakes, post about those too please!

I tested my bot against the test positions and I think it went pretty well. There were a handful of positions that had 9 rabbits on a side, which upset my bot (I optimised speed over flexibility). And I think there is a problem with position 393.atp.

If you're willing to put these on github (or if you won't object to me putting it up there) I'd be happy to send you some pull requests for things I find.