Arimaa Forum - Print Page


    
      
        Arimaa Forum
        (http://arimaa.com/arimaa/forum/cgi/YaBB.cgi)
      

        Arimaa >> Bot Development >> Regression testing
        
(Message started by: doublep on Oct 1^st, 2009, 1:32pm)

Title: Regression testing
Post by doublep on Oct 1^st, 2009, 1:32pm

I'm curious if any of the developers uses regression testing. It seems not that useful for strategic things, i.e. evaluating whole board positions, but I find it invaluable for tactical searches, like capturing or finding win in one move. Strategical tests could be possible in cases of only one or few "correct" moves (i.e. if you don't make this move, you are inevitably losing a major piece, like that), but I'm not a good enough player to spot this.

Badger uses a slightly modified version of GTP (http://www.lysator.liu.se/~gunnar/gtp/), so I'm able to formulate regression problems like this:

Code:

# Pull opponent ahead.
clear_board
setup_squares Ra7 Cb8 ra8 ec7 cb6
50010 find_one_move_win white
#? [true Cb8s-ra8e-Ra7n]

The last line (that starting with #?) is just a comment from engine point of view, but for regression script it is a marker that provides expected answer to the last command. I.e. the script considers test passed if Badger's answer to 'find_one_move_win white' in that position is 'true Cb8s-ra8e-Ra7n'.

Eventually I even wrote automated tools to find mistakes in various tactical searches, both false positives (i.e. invalid moves or moves that don't yield results the engine claims they would) and missed problem solutions (missed one move wins, missed captures). The tool also interfaces with Badger through GTP. For its use, Badger provides two solvers for all tactical problems it handles: the production solver --- very fast, the one that is tested, --- and "robust" solver --- slow, simple, mostly brute-force and complete (unless there are unspotted bugs, of course ;). Minor custom editing is required, but most work is automatic. Tests generated with such a tool look this way:

Code:

setup_board -----r----r--cr--m-h--rr-H-h--r-r---Ed-----DC-eM-----CDdR-R----R
40 capture_anything black
#? [!true hd5s-Dd3w-Dc3x-hd4s]

Do any other bots use anything similar?

Title: Re: Regression testing
Post by Janzert on Oct 2^nd, 2009, 1:56pm

For general testing I have a set of ~100 board positions mostly just drawn from games where the bot made an obviously bad move in the game. I just look to see that the differences with the new version at least seem reasonable.

For the static goal and capture search I used fuzz testing¹ to find the errors and missed cases.

Janzert

¹ Check random positions and make sure the static search finds the same results as the full move generation does