A checker is an external script which is called from the main testsuite program to verify the correctness of a specific test. More than one checker can be applied to individual test. For example, the user might want to check whether the run was successful, whether output was produced and whether results are within specified thresholds. The checkers can be written in any scripting language (bash, python ...). Communication of runtime variables between the testsuite and a checker is achieved via environment variables. A checker can access a set of environment variables defined by the teststuite.
TS_BASEDIR
Root directory of the testsuite.
TS_VERBOSE
Verbosity level requested by the user for running the testsuite.
TS_RUNDIR
Directory where the current test was run.
TS_LOGFILE
File which contains the standard output and standard error of the current test.
TS_NAMELISTDIR
Directory containing the namelist of the current test.
TS_REFOUTDIR
Directory containing the reference files for the current test.
Each checker should return an exit code with the following definition:
0
MATCH
Results match bit-by-bit as compared to the reference.
10
OK
Results match the reference within defined thresholds.
15
SKIP
Test not applicable and thus skipped.
20
FAIL
Test failed, results are outside of thresholds.
30
CRASH
Test failed due to model crash.
The checkers to be called for a given test are defined within the testlist.xml file. All checkers are called irrespective of the exit code of preceding tests. The overall status of a test is defined as the maximum of the exit codes of all checkers called.
When bit identical results are expected, as for example when comparing two different parallelizations on a given system, the identical_check.py checker should be called. This checker expects a YUPRTEST output file and absolute differences between the output of the test and the reference file are compared to within a zero tolerance threshold. Note that bit identical results can only be expected when two simulations are run on the same system. A test using the identical check.py should always have reference directory pointing to a previous test in the working folder, e.g. ../test_1 .