Pruebas automatizadas de binarios de SVN en HPC

Question

Me gustaría configurar algunas pruebas automatizadas de casos de prueba en binarios Fortran dentro de un repositorio SVN, bajo demanda o con un trabajo Cron. Para complicar un poco el problema, los casos de prueba se ejecutarían en un clúster computacional, de modo que cada caso de prueba necesitaría generar un script PBS. (Entorno Linux)

Hay muchas pruebas web y soluciones de pruebas unitarias, sin embargo, no pude encontrar nada para probar los binarios directamente. Digamos, proporcione entradas y compare las salidas con la solución esperada.

¿Alguna sugerencia sobre cómo debería abordarse esto?

Answer 1

I agree that this is something that would be pretty straightforward to script. Depending on how your binaries are configured and how different your programs are from one another, you could even include the testing scripts as part of your SVN repository.

Since you're in a batch system, you have some freedom for scheduling the tests. You may want to have a generic "verify" job that you can provide setup parameters to (e.g. to locations of expected and actual program output output). Nick mentioned using grep to check the output of qstat for your job ID, but you can tell PBS to hold a job until another job completes. This would mean that you have something like:

...
#PBS -N run_test_1
#PBS -a 200906270000
...
<compile test>
<run test>
<save output>

when submitted, save the job ID returned by qsub (how you do this is dependent on your platform - usually something like job_id=$(qsub $JOB_FILE) is sufficient). Then, plug that value in to another script:

...
#PBS -N verify_test_1
#PBS -W depend=afterany:$job_id
<run comparison>
...

This will (when the proper value of job_id is inserted) hold the execution of the test run until midnight on June 27, 2009, and hold the execution of the verification job until the test job completes (the afterany directive indicates that it should always run after the first job - not just if it's successful).

Depending on your cluster's turnaround time, though, you could put all of this in one script, though you could still use the PBS time-based holds to only run at a particular time. I've recently started using Python instead of shell scripts to handle even this system-related jobs - I tested yesterday making the Python script executable and adding the PBS directives straight into the source - it seemed to work very well.

Answer 2

There may be a better answer that is more pre-packaged, but I believe that Buildbot is configurable enough that as long as you can write a python script to run your binary, it should work for what you want.

http://buildbot.net/

Answer 3

It seems like for simple input/output testing you could knock up a script yourself to do this...

e.g.

for each program
    compile
    submit to queue
    wait for finish
    check output

In practice, you'd probably want to submit more than one job to the queue, but the idea is the same.

Here are a few ideas off the top of my head on how you could do each step.

Submitting

You could use a template PBS script, and use sed to find/replace intput tokens with input values for the program before submitting

Waiting for finish

You could repeatedly grep the output of qstat for your job id, to wait for it to finish

Check output

Check the job's output file against expected output. You could either save the expected output to a file and use diff, or have a list of regular expression that must match for each run