William E. Hart's Blog

Software testing is widely recognized as a best practice for software development . Software tests define expected functionality, and they can focus developer efforts by providing an objective assessment of the state of a software project. Additionally, software testing data can provide evidence that a software package can be reliably used. For example, when evaluating whether to try out open source software, I routinely look for software testing data to confirm which platforms the software will run on, the versions of associated software that is used, and test coverage statistics that indicate how much of the the code is tested. Unfortunately, most open source software projects do not publish software test data.  I suspects that this indicates that a small fraction of OSS projects have robust test suites.  However, this also reflects another aspect of the OSS community:  hosting facilities for open source software do not support web-based testing facilities, like Jenk...

The role of the FAST software repository has evolved over the past few years, and it is time to rethink what the goal of FAST is. See Rethinking the goal of FAST for some perspective about FAST. The upshot is that FAST has been reorganized to support an ensemble of independent software packages related to agile software development. One important change in this reorganization has been the decomposition of the FAST Python software into independent components. Some elements of FAST are deprecated, and the rest have been spread out into various packages. See the FAST Blog for further details.

The gcovr command provides a utility for running the gcov command and summarizing code coverage results. This command is inspired by the Python coverage.py package, which provides a similar utility in Python. Further, gcovr can be viewed as a command-line alternative of the lcov utility, which runs gcov and generates an HTML output. Recently, the gcovr script has been broken out into a separate software package, which is managed at https://software.sandia.gov/svn/public/fast/gcovr This package has been uploaded to the Python PyPI repository to facilitate it installation with easy_install: easy_install gcovr Alternatively, the gcovr script can be downloaded directly: https://software.sandia.gov/svn/public/fast/gcovr/trunk/scripts/gcovr See the gcovr Trac page for further details about this tool. Although gcovr is increasingly used to generate coverage statistics within Hudson, the gcovr wiki documents a command-line text summary that I personally find very useful when developing...

A while back I developed EXACT , a Python package for executing computational experiments using an XML-defined process. EXACT was designed to fill a particular niche in software testing: performing computational tests that involve the application of solvers to a suite of test problems. This sort of testing arises a lot when doing functionality testing for scientific software. Unfortunately, EXACT was too complex: The XML specification was complex and difficult to read Experiments with many factors were assigned generic experiment IDs It was hard to replication the execution of specific experiments The experimental results were captured in XML results files that were difficult to browse Even my close collaborators struggled to setup, run and analyze computational experiments! {sigh} I have recently developed the pyutilib.autotest Python package to provide a simpler alternative to EXACT. This package uses a YAML test configuration file to specify the solvers and problems that are ...

At PyCon 2009, Aaron Maxwell gave a presentation about the use of BuildBot to support an automated software QA infrastructure. Listening to his talk ( online ) made me think more carefully about the reasons I am not using BuildBot, which I took a look at several years ago.  After working with a custom automated build tool for a few years, I have recently begun using Hudson to automate software quality processes for a variety of open source software packages.  Hudson automates the following QA activities for these packages: portability tests - building packages with different compilers, language versions and compute platforms continuous integration - rapid builds and software tests to provide developers continuous feedback integration tests - builds that test the integration of different software tools archiving QA statistics - test histories, code coverage statistics, build times, etc. managing third-party builds - building third-party libraries that my codes depend on Althou...

The gcovr command provides a utility for running the gcov command and summarizing code coverage results. This command is inspired by the Python coverage.py package, which provides a similar utility in Python. Further, gcovr can be viewed as a command-line alternative of the lcov utility, which runs gcov and generates an HTML output. The gcovr command currently generates two different types of output: Text Summary For each file that generates gcov statistics, gcovr will summarize the number of lines covered, the percentage of coverage and enumerate the lines that are not covered. Cobertura XML An XML summary of the coverage statistics can be generated in a format that is consistent with Cobertura. I find the text summary quite convenient for interactive assessment of coverage, especially as I design tests to improve coverage. The Cobertura summary can be used by continues build tools like Hudson . For example, see the acro-utilib coverage report that was generated with gcovr, ...

There are many tools available that can be used to monitor memory usage in computer programs. However, there are few tools that can be applied to monitor the memory usage of a specific process in an automated manner. Most memory monitoring tools provide a gui that a user can monitor. However, these tools are not useful in contexts where memory must be monitored repeatedly. For example, automated software tests may require checks to validate that the memory usage does not exceed expected limits. The memmon command is a new memory monitoring tool that is included in the UTILIB software library. memmon provides a convenient mechanism to report the maximum amount of memory that a process uses. The memmon command requires the absolute path to the command that will be executed. Beyond that, its default syntax is quite simple: $ ./memmon /bin/sleep 1 53768 Kb used The memmon command can also be used to terminate a process whose memory exceeds a specified threshold: $ ./memmon -k 10 /bin/...

There are many applications where you want to apply a code to a variety of data sets, and verify that you get the correct output. In this context, what you want is a test generator, which can dynamically create tests, based on the set of data sets that are available for testing. Unfortunately, this does not appear to be a feature of unittest . The closest I have seen to this, is the support for test generators in the nose package, which extends unittest to provide test discovery mechanisms. However, that test generation feature is somewhat limited; it only applies to test functions that are Python generators, and not to similar class methods. The following example shows how to directly insert new test methods into a unittest.TestCase class : # # A simple example for generating tests in the Python unittest framework # import glob import unittest # # Defining the class that will contain the new tests # class TestCases(unittest.TestCase): pass # # A generic function that performs a te...