A New Python Package: pyutilib.autotest

A while back I developed EXACT, a Python package for executing computational experiments using an XML-defined process. EXACT was designed to fill a particular niche in software testing: performing computational tests that involve the application of solvers to a suite of test problems. This sort of testing arises a lot when doing functionality testing for scientific software.

Unfortunately, EXACT was too complex:

• The XML specification was complex and difficult to read
• Experiments with many factors were assigned generic experiment IDs
  
• It was hard to replication the execution of specific experiments
• The experimental results were captured in XML results files that were difficult to browse

Even my close collaborators struggled to setup, run and analyze computational experiments! {sigh}

I have recently developed the pyutilib.autotest Python package to provide a simpler alternative to EXACT. This package uses a YAML test configuration file to specify the solvers and problems that are exercised in a test. The tests setup and executed with Python's unittest package, which allows the user to apply a rich set of testing methods. The user can specify a test driver in Python that defines how each test is executed, using the solver and problem options that are specified in the configuration file.

Here's a simple example of a YAML test specification, where the goal is to apply the UNIX cat command with different options to different test files:

driver: example

solvers:
    cat:
    cat2:
        name: cat
        cat_options: -n

problems:
    f1:
        file: file1.txt
    f2:
        file: file2.txt

suites:
    suite1:
        solvers:
            cat:
            cat2:
        problems:
            f1:
            f2:

See the pyutilib.autotest PyPI documentation for further details.