Welcome, Guest. Please Login or Register.
Nov 22nd, 2024, 11:46am

Home Home Help Help Search Search Members Members Login Login Register Register
Arimaa Forum « Goal Test Positions »


   Arimaa Forum
   Arimaa
   Bot Development
(Moderator: supersamu)
   Goal Test Positions
« Previous topic | Next topic »
Pages: 1 2  Reply Reply Notify of replies Notify of replies Send Topic Send Topic Print Print
   Author  Topic: Goal Test Positions  (Read 9631 times)
rabbits
Forum Guru
*****




Arimaa player #1337

   
WWW

Gender: male
Posts: 108
Goal Test Positions
« on: Nov 5th, 2011, 12:43am »
Quote Quote Modify Modify

When I was working on the static goal search for my bot a year and a half ago, I hand crafted a bunch of test positions.  While skimming the chat archive the other day, I saw some talk about test positions, so I figured I might as well share mine.
 
Goal Test Positions
 
Let me know if anyone actually tests their bot with these positions.  Also, if there are any mistakes, post about those too please!
IP Logged

rbarreira
Forum Guru
*****



Arimaa player #1621

   


Gender: male
Posts: 605
Re: Goal Test Positions
« Reply #1 on: Nov 5th, 2011, 4:22am »
Quote Quote Modify Modify

Thanks for posting those! I definitely want to try them out but I'll need some automatic script to run my bot with them (and check the answer), as they're so many.
 
Why do some positions have "cangoal true" and some others have "cangoal [1..4]"? Does it simply mean you didn't bother filling in how many steps the goal consists of?
« Last Edit: Nov 5th, 2011, 4:25am by rbarreira » IP Logged
rabbits
Forum Guru
*****




Arimaa player #1337

   
WWW

Gender: male
Posts: 108
Re: Goal Test Positions
« Reply #2 on: Nov 5th, 2011, 12:12pm »
Quote Quote Modify Modify

Of course you would want to automate the testing!  I had my bot setup so that if I passed it the --atp flag it would run these arimaa test positions.
 
I think I started out with only "cangoal true" and "cangoal false" and then added the steps needed option without changing my older tests.
 
That reminds me, I wrote a gedit text highlighting thing for Arimaa positions.  I'll share that too Smiley
 
GTK Sourceview Arimaa color scheme / ATP language
 
Here is a screenshot of what it looks like on my computer right now:
IP Logged

rbarreira
Forum Guru
*****



Arimaa player #1621

   


Gender: male
Posts: 605
Re: Goal Test Positions
« Reply #3 on: Nov 5th, 2011, 12:22pm »
Quote Quote Modify Modify

I managed to run all the positions.
 
There were no false positives (i.e. the positions with "cangoal false" all worked as they should). This is good as false positives are worse than false negatives.
 
As I expected, there were plenty of false negatives, even more than I expected - 130 of the 303 goal positions reported no goal found. I'll be going through them and adding cases to my bot (hopefully some of those cases are redundant and I don't have to add 130 pieces of code). I'll let you know if I find any mistakes
 
Thanks again for the test positions!
IP Logged
rabbits
Forum Guru
*****




Arimaa player #1337

   
WWW

Gender: male
Posts: 108
Re: Goal Test Positions
« Reply #4 on: Nov 5th, 2011, 12:36pm »
Quote Quote Modify Modify

You had tested your static goal search against all of the games in the Arimaa games database, right?  I guess not all goaling patterns have actually occurred in real games then!
IP Logged

rbarreira
Forum Guru
*****



Arimaa player #1621

   


Gender: male
Posts: 605
Re: Goal Test Positions
« Reply #5 on: Nov 5th, 2011, 12:49pm »
Quote Quote Modify Modify

My bot does not detect all goals in the Arimaa games database either. There are more than 600 games (out of around 120,000 ended in goal) where goal is not found.
 
That said I wouldn't be surprised if your test cases have some goals which didn't happen yet, given the high rate of not found goals I have!
« Last Edit: Nov 5th, 2011, 12:49pm by rbarreira » IP Logged
UruramTururam
Forum Guru
*****




Arimaa player #2537

   
WWW

Gender: male
Posts: 319
Re: Goal Test Positions
« Reply #6 on: Nov 5th, 2011, 4:24pm »
Quote Quote Modify Modify

on Nov 5th, 2011, 12:12pm, rabbits wrote:

Here is a screenshot of what it looks like on my computer right now:

 
Is there a forced goal here? Wow...
IP Logged

Caffa et bucella per attactionem corporum venit ad stomachum meum.
BGG Arimaa badges - get your own one!
Swynndla
Forum Guru
*****



Arimaa player #1821

   


Posts: 235
Re: Goal Test Positions
« Reply #7 on: Nov 5th, 2011, 9:37pm »
Quote Quote Modify Modify

Thanks for sharing rabbits. Smiley
 
I installed gedit (as I don't use gnome) but it didn't work - probably because I installed gedit version 2 not 3? (as version 2 is available in debian stable but not 3) ... I copied the files to 2.0 directories instead of the (non-existent) 3.0 directories, but it still would work, eg:
Code:
(gedit:8896): GtkSourceView-WARNING **: in file /usr/share/gtksourceview-2.0/language-specs/atp.lang: style 'atp:board' not defined

etc
 
No big deal - I'm happy to wait until version 3 is in debian stable.  Smiley
IP Logged
lightvector
Forum Guru
*****



Arimaa player #2543

   


Gender: male
Posts: 197
Re: Goal Test Positions
« Reply #8 on: Nov 5th, 2011, 10:01pm »
Quote Quote Modify Modify

Nice set of test positions. I'll be trying sharp on them soon too. Thanks for posting them.
 
Rabbits: It actually is the case that being correct on the entire Arimaa game database is not sufficient to have a correct goal search. Some goaling patterns actually haven't occurred in real games! In earlier testing, a few months ago, I found that even after being correct on the entire game database, there were still a few types of goals that sharp missed.
 
The way I found these additional missed patterns was to not only test each position in the game database, but also a hundred or so perturbations of each position. Each perturbation took the original position and modified it by randomly sprinkling some additional rabbits and other pieces around the board, possibly overwriting existing pieces. The idea was to transform typical midgame positions into strenuous goal threat tests. This frequently happens when an enemy rabbit gets randomly placed behind your lines, or when an existing advanced rabbit gets an additional supporting piece dropped next to it, and in either case, possibly overwriting a defending piece too. Perturbing the position also worked well to detect bugs in positions where a rabbit in the original position could almost goal or barely goal, so that the perturbations would readily flip the status back and forth in varied ways.
 
 
 
« Last Edit: Nov 5th, 2011, 10:23pm by lightvector » IP Logged
rabbits
Forum Guru
*****




Arimaa player #1337

   
WWW

Gender: male
Posts: 108
Re: Goal Test Positions
« Reply #9 on: Nov 6th, 2011, 12:17pm »
Quote Quote Modify Modify

UruramTururam: Haha, no there is not a forced goal in the position in the screenshot!  That position is not included in the goal test positions that I shared.  It's just a position I used to check that my bot was generating the correct number of legal steps and moves.
 
Swynndla: I actually just changed the install script to install in the GtkSourceView version 3.0 directory yesterday when I reinstalled those files, but I didn't change the files.  If you look inside of the files, you will see that they are not version 3.0.  As for the warnings, well, I had never noticed before, but I get those too:
Code:
(gedit:18608): GtkSourceView-WARNING **: in file /usr/share/gtksourceview-3.0/language-specs/atp.lang: style 'atp:board' not defined
 
(gedit:18608): GtkSourceView-WARNING **: in file /usr/share/gtksourceview-3.0/language-specs/atp.lang: style 'atp:board-labels' not defined
 
(gedit:18608): GtkSourceView-WARNING **: in file /usr/share/gtksourceview-3.0/language-specs/atp.lang: style 'atp:board-inside' not defined
 
(gedit:18608): GtkSourceView-WARNING **: in file /usr/share/gtksourceview-3.0/language-specs/atp.lang: style 'atp:gold' not defined
 
(gedit:18608): GtkSourceView-WARNING **: in file /usr/share/gtksourceview-3.0/language-specs/atp.lang: style 'atp:silver' not defined

Hmmmm, oops!  It still works for me though...
 
By the way, is Debian still using Gnome2?  If so, I might switch to it!
 
Lightvector: Cool, thanks for verifying that!  I created a lot of these positions by perturbing other positions too, except that I did so manually and with the motivation to test each chunk of code as I wrote it.
IP Logged

Swynndla
Forum Guru
*****



Arimaa player #1821

   


Posts: 235
Re: Goal Test Positions
« Reply #10 on: Nov 6th, 2011, 1:40pm »
Quote Quote Modify Modify

Oh silly me - I don't know anything about gedit ... I had to go to Preferences > Fonts & Colors  and then choose Arimaa ... now it works. Smiley
IP Logged
rabbits
Forum Guru
*****




Arimaa player #1337

   
WWW

Gender: male
Posts: 108
Re: Goal Test Positions
« Reply #11 on: Nov 6th, 2011, 1:43pm »
Quote Quote Modify Modify

Yeah, I had to make a whole "style scheme" to get the gold, silver, and brown colors I wanted...
IP Logged

Swynndla
Forum Guru
*****



Arimaa player #1821

   


Posts: 235
Re: Goal Test Positions
« Reply #12 on: Nov 6th, 2011, 1:48pm »
Quote Quote Modify Modify

I think debian stable & testing uses gnome version 2.30 ... that's what the repo's tell me anyway?  I use KDE and LXDE so I don't know much about gnome.  By going to debian stable or testing, some packages wont be as up-to-date as with the latest ubuntu.
IP Logged
doublep
Forum Guru
*****



Badger author

   


Gender: male
Posts: 82
Re: Goal Test Positions
« Reply #13 on: Nov 11th, 2011, 2:37pm »
Quote Quote Modify Modify

I can share regression files from Badger if anyone wants, but they are coded in GTP (http://www.lysator.liu.se/~gunnar/gtp/) reworded for Arimaa.  So a test looks like:
 
clear_board
setup_squares Ra7 Cb8 ra8 ec7 cb6
50010 find_one_move_win white
#? [true Cb8s-ra8e-Ra7n]
 
Or one found with automatic error-finding tool:
 
setup_board ----r---rrrMdr-rrReR----DE------c-----m-------hr---------RR-RRRR
670 find_one_move_win white
#? [true rc7n-ec6x-Md7w-Rd6n-Rd7n]
 
I also don't have anywhere to host files :/
IP Logged
Dolus
Forum Guru
*****




Arimaa player #6845

    DolusDeceit


Gender: male
Posts: 86
Re: Goal Test Positions
« Reply #14 on: Nov 11th, 2011, 3:23pm »
Quote Quote Modify Modify

That sounds helpful.
 
As far as sharing files, I expect somebody should have a good solution to this, but otherwise, we could set up a Dropbox account or something along those lines.
IP Logged

Pages: 1 2  Reply Reply Notify of replies Notify of replies Send Topic Send Topic Print Print

« Previous topic | Next topic »

Arimaa Forum » Powered by YaBB 1 Gold - SP 1.3.1!
YaBB © 2000-2003. All Rights Reserved.