Monday, June 28, 2010

A New Python Package: pyutilib.autotest

A while back I developed EXACT, a Python package for executing computational experiments using an XML-defined process. EXACT was designed to fill a particular niche in software testing: performing computational tests that involve the application of solvers to a suite of test problems. This sort of testing arises a lot when doing functionality testing for scientific software.

Unfortunately, EXACT was too complex:
  • The XML specification was complex and difficult to read
  • Experiments with many factors were assigned generic experiment IDs
  • It was hard to replication the execution of specific experiments
  • The experimental results were captured in XML results files that were difficult to browse
Even my close collaborators struggled to setup, run and analyze computational experiments! {sigh}

I have recently developed the pyutilib.autotest Python package to provide a simpler alternative to EXACT. This package uses a YAML test configuration file to specify the solvers and problems that are exercised in a test. The tests setup and executed with Python's unittest package, which allows the user to apply a rich set of testing methods. The user can specify a test driver in Python that defines how each test is executed, using the solver and problem options that are specified in the configuration file.

Here's a simple example of a YAML test specification, where the goal is to apply the UNIX cat command with different options to different test files:
driver: example

solvers:
cat:
cat2:
name: cat
cat_options: -n

problems:
f1:
file: file1.txt
f2:
file: file2.txt

suites:
suite1:
solvers:
cat:
cat2:
problems:
f1:
f2:
See the pyutilib.autotest PyPI documentation for further details.

No comments:

Post a Comment