Unfortunately, EXACT was too complex:
- The XML specification was complex and difficult to read
- Experiments with many factors were assigned generic experiment IDs
- It was hard to replication the execution of specific experiments
- The experimental results were captured in XML results files that were difficult to browse
I have recently developed the pyutilib.autotest Python package to provide a simpler alternative to EXACT. This package uses a YAML test configuration file to specify the solvers and problems that are exercised in a test. The tests setup and executed with Python's unittest package, which allows the user to apply a rich set of testing methods. The user can specify a test driver in Python that defines how each test is executed, using the solver and problem options that are specified in the configuration file.
Here's a simple example of a YAML test specification, where the goal is to apply the UNIX cat command with different options to different test files:
driver: exampleSee the pyutilib.autotest PyPI documentation for further details.