Author |
Topic: Goal Test Positions (Read 9631 times) |
|
rabbits
Forum Guru
Arimaa player #1337
Gender:
Posts: 108
|
|
Goal Test Positions
« on: Nov 5th, 2011, 12:43am » |
Quote Modify
|
When I was working on the static goal search for my bot a year and a half ago, I hand crafted a bunch of test positions. While skimming the chat archive the other day, I saw some talk about test positions, so I figured I might as well share mine. Goal Test Positions Let me know if anyone actually tests their bot with these positions. Also, if there are any mistakes, post about those too please!
|
|
IP Logged |
|
|
|
rbarreira
Forum Guru
Arimaa player #1621
Gender:
Posts: 605
|
|
Re: Goal Test Positions
« Reply #1 on: Nov 5th, 2011, 4:22am » |
Quote Modify
|
Thanks for posting those! I definitely want to try them out but I'll need some automatic script to run my bot with them (and check the answer), as they're so many. Why do some positions have "cangoal true" and some others have "cangoal [1..4]"? Does it simply mean you didn't bother filling in how many steps the goal consists of?
|
« Last Edit: Nov 5th, 2011, 4:25am by rbarreira » |
IP Logged |
|
|
|
rabbits
Forum Guru
Arimaa player #1337
Gender:
Posts: 108
|
|
Re: Goal Test Positions
« Reply #2 on: Nov 5th, 2011, 12:12pm » |
Quote Modify
|
Of course you would want to automate the testing! I had my bot setup so that if I passed it the --atp flag it would run these arimaa test positions. I think I started out with only "cangoal true" and "cangoal false" and then added the steps needed option without changing my older tests. That reminds me, I wrote a gedit text highlighting thing for Arimaa positions. I'll share that too GTK Sourceview Arimaa color scheme / ATP language Here is a screenshot of what it looks like on my computer right now:
|
|
IP Logged |
|
|
|
rbarreira
Forum Guru
Arimaa player #1621
Gender:
Posts: 605
|
|
Re: Goal Test Positions
« Reply #3 on: Nov 5th, 2011, 12:22pm » |
Quote Modify
|
I managed to run all the positions. There were no false positives (i.e. the positions with "cangoal false" all worked as they should). This is good as false positives are worse than false negatives. As I expected, there were plenty of false negatives, even more than I expected - 130 of the 303 goal positions reported no goal found. I'll be going through them and adding cases to my bot (hopefully some of those cases are redundant and I don't have to add 130 pieces of code). I'll let you know if I find any mistakes Thanks again for the test positions!
|
|
IP Logged |
|
|
|
rabbits
Forum Guru
Arimaa player #1337
Gender:
Posts: 108
|
|
Re: Goal Test Positions
« Reply #4 on: Nov 5th, 2011, 12:36pm » |
Quote Modify
|
You had tested your static goal search against all of the games in the Arimaa games database, right? I guess not all goaling patterns have actually occurred in real games then!
|
|
IP Logged |
|
|
|
rbarreira
Forum Guru
Arimaa player #1621
Gender:
Posts: 605
|
|
Re: Goal Test Positions
« Reply #5 on: Nov 5th, 2011, 12:49pm » |
Quote Modify
|
My bot does not detect all goals in the Arimaa games database either. There are more than 600 games (out of around 120,000 ended in goal) where goal is not found. That said I wouldn't be surprised if your test cases have some goals which didn't happen yet, given the high rate of not found goals I have!
|
« Last Edit: Nov 5th, 2011, 12:49pm by rbarreira » |
IP Logged |
|
|
|
UruramTururam
Forum Guru
Arimaa player #2537
Gender:
Posts: 319
|
|
Re: Goal Test Positions
« Reply #6 on: Nov 5th, 2011, 4:24pm » |
Quote Modify
|
on Nov 5th, 2011, 12:12pm, rabbits wrote: Here is a screenshot of what it looks like on my computer right now: |
| Is there a forced goal here? Wow...
|
|
IP Logged |
Caffa et bucella per attactionem corporum venit ad stomachum meum. BGG Arimaa badges - get your own one!
|
|
|
Swynndla
Forum Guru
Arimaa player #1821
Posts: 235
|
|
Re: Goal Test Positions
« Reply #7 on: Nov 5th, 2011, 9:37pm » |
Quote Modify
|
Thanks for sharing rabbits. I installed gedit (as I don't use gnome) but it didn't work - probably because I installed gedit version 2 not 3? (as version 2 is available in debian stable but not 3) ... I copied the files to 2.0 directories instead of the (non-existent) 3.0 directories, but it still would work, eg: Code:(gedit:8896): GtkSourceView-WARNING **: in file /usr/share/gtksourceview-2.0/language-specs/atp.lang: style 'atp:board' not defined |
| etc No big deal - I'm happy to wait until version 3 is in debian stable.
|
|
IP Logged |
|
|
|
lightvector
Forum Guru
Arimaa player #2543
Gender:
Posts: 197
|
|
Re: Goal Test Positions
« Reply #8 on: Nov 5th, 2011, 10:01pm » |
Quote Modify
|
Nice set of test positions. I'll be trying sharp on them soon too. Thanks for posting them. Rabbits: It actually is the case that being correct on the entire Arimaa game database is not sufficient to have a correct goal search. Some goaling patterns actually haven't occurred in real games! In earlier testing, a few months ago, I found that even after being correct on the entire game database, there were still a few types of goals that sharp missed. The way I found these additional missed patterns was to not only test each position in the game database, but also a hundred or so perturbations of each position. Each perturbation took the original position and modified it by randomly sprinkling some additional rabbits and other pieces around the board, possibly overwriting existing pieces. The idea was to transform typical midgame positions into strenuous goal threat tests. This frequently happens when an enemy rabbit gets randomly placed behind your lines, or when an existing advanced rabbit gets an additional supporting piece dropped next to it, and in either case, possibly overwriting a defending piece too. Perturbing the position also worked well to detect bugs in positions where a rabbit in the original position could almost goal or barely goal, so that the perturbations would readily flip the status back and forth in varied ways.
|
« Last Edit: Nov 5th, 2011, 10:23pm by lightvector » |
IP Logged |
|
|
|
rabbits
Forum Guru
Arimaa player #1337
Gender:
Posts: 108
|
|
Re: Goal Test Positions
« Reply #9 on: Nov 6th, 2011, 12:17pm » |
Quote Modify
|
UruramTururam: Haha, no there is not a forced goal in the position in the screenshot! That position is not included in the goal test positions that I shared. It's just a position I used to check that my bot was generating the correct number of legal steps and moves. Swynndla: I actually just changed the install script to install in the GtkSourceView version 3.0 directory yesterday when I reinstalled those files, but I didn't change the files. If you look inside of the files, you will see that they are not version 3.0. As for the warnings, well, I had never noticed before, but I get those too: Code:(gedit:18608): GtkSourceView-WARNING **: in file /usr/share/gtksourceview-3.0/language-specs/atp.lang: style 'atp:board' not defined (gedit:18608): GtkSourceView-WARNING **: in file /usr/share/gtksourceview-3.0/language-specs/atp.lang: style 'atp:board-labels' not defined (gedit:18608): GtkSourceView-WARNING **: in file /usr/share/gtksourceview-3.0/language-specs/atp.lang: style 'atp:board-inside' not defined (gedit:18608): GtkSourceView-WARNING **: in file /usr/share/gtksourceview-3.0/language-specs/atp.lang: style 'atp:gold' not defined (gedit:18608): GtkSourceView-WARNING **: in file /usr/share/gtksourceview-3.0/language-specs/atp.lang: style 'atp:silver' not defined |
| Hmmmm, oops! It still works for me though... By the way, is Debian still using Gnome2? If so, I might switch to it! Lightvector: Cool, thanks for verifying that! I created a lot of these positions by perturbing other positions too, except that I did so manually and with the motivation to test each chunk of code as I wrote it.
|
|
IP Logged |
|
|
|
Swynndla
Forum Guru
Arimaa player #1821
Posts: 235
|
|
Re: Goal Test Positions
« Reply #10 on: Nov 6th, 2011, 1:40pm » |
Quote Modify
|
Oh silly me - I don't know anything about gedit ... I had to go to Preferences > Fonts & Colors and then choose Arimaa ... now it works.
|
|
IP Logged |
|
|
|
rabbits
Forum Guru
Arimaa player #1337
Gender:
Posts: 108
|
|
Re: Goal Test Positions
« Reply #11 on: Nov 6th, 2011, 1:43pm » |
Quote Modify
|
Yeah, I had to make a whole "style scheme" to get the gold, silver, and brown colors I wanted...
|
|
IP Logged |
|
|
|
Swynndla
Forum Guru
Arimaa player #1821
Posts: 235
|
|
Re: Goal Test Positions
« Reply #12 on: Nov 6th, 2011, 1:48pm » |
Quote Modify
|
I think debian stable & testing uses gnome version 2.30 ... that's what the repo's tell me anyway? I use KDE and LXDE so I don't know much about gnome. By going to debian stable or testing, some packages wont be as up-to-date as with the latest ubuntu.
|
|
IP Logged |
|
|
|
doublep
Forum Guru
Badger author
Gender:
Posts: 82
|
|
Re: Goal Test Positions
« Reply #13 on: Nov 11th, 2011, 2:37pm » |
Quote Modify
|
I can share regression files from Badger if anyone wants, but they are coded in GTP (http://www.lysator.liu.se/~gunnar/gtp/) reworded for Arimaa. So a test looks like: clear_board setup_squares Ra7 Cb8 ra8 ec7 cb6 50010 find_one_move_win white #? [true Cb8s-ra8e-Ra7n] Or one found with automatic error-finding tool: setup_board ----r---rrrMdr-rrReR----DE------c-----m-------hr---------RR-RRRR 670 find_one_move_win white #? [true rc7n-ec6x-Md7w-Rd6n-Rd7n] I also don't have anywhere to host files :/
|
|
IP Logged |
|
|
|
Dolus
Forum Guru
Arimaa player #6845
Gender:
Posts: 86
|
|
Re: Goal Test Positions
« Reply #14 on: Nov 11th, 2011, 3:23pm » |
Quote Modify
|
That sounds helpful. As far as sharing files, I expect somebody should have a good solution to this, but otherwise, we could set up a Dropbox account or something along those lines.
|
|
IP Logged |
|
|
|
|