Using Open-Source Tools to Manage Software Quality

At PyCon 2009, Aaron Maxwell gave a presentation about the use of BuildBot to support an automated software QA infrastructure. Listening to his talk (online) made me think more carefully about the reasons I am not using BuildBot, which I took a look at several years ago.  After working with a custom automated build tool for a few years, I have recently begun using Hudson to automate software quality processes for a variety of open source software packages.  Hudson automates the following QA activities for these packages:
  • portability tests - building packages with different compilers, language versions and compute platforms
  • continuous integration - rapid builds and software tests to provide developers continuous feedback
  • integration tests - builds that test the integration of different software tools
  • archiving QA statistics - test histories, code coverage statistics, build times, etc.
  • managing third-party builds - building third-party libraries that my codes depend on
Although I am reasonably happy with Hudson, I must admit that I did not immediately decide that it was perfect for my needs the first time I looked at it.  However, as the scope of my QA needs has grown, it has become critical to have a flexible, extensible strategy for automating software QA activities. The following high-level issues have proven to be major considerations when assessing the viability of tools like BuildBot and Hudson:
  • GUI/web interface
    GUI and web interfaces are key to ensuring that developers regularly use the QA data that is being generated. Interactive interrogation of QA current data facilitates effective use of this data, and GUI interfaces are very important when developers do not all have access to the same computing platforms. These interfaces can also convey valuable QA in a concise manner, such as graphical representations of QA history (e.g test failure/successes of time).

  • Extensibility
    Any automation framework is going to need to be customized to adapt to local operating constraints. Thus, the extensibility of the automation framework is a key issue for end-users. A particularly effective strategy for supporting this flexibility is with plugin components, which are supported in Hudson.

  • Loosely Coupled QA Tools
    Hudson uses a standards-based approach for integrating QA information. QA activities can be initiated in a very generic manner, using shell commands whose scope is not restricted. If the QA information is provided in a standard format, then Hudson can integrate it into its QA summaries. For example, Hudson recognizes testing results that are documented with the xUnit XML format, and code coverage results that are documented with the Cobertura XML format. This strategy supports a loose coupling between the QA processes and the use of Hudson, which allows for the application of a heterogeneous set of QA tools, including tools for different test harnesses and programming languages!

  • Compute Resource Management
    Coordinating of a large number of QA activities requires scalable strategies for managing computing resources. Frameworks like Hudson provide basic resource management strategies, including dynamic scheduling of continuous integration builds on a build farm. More generally, scalable automated testing tools need to support strategies like fractional factorial test designs, which test many build options (configuration, platform, compiler, etc) with a small number of builds. Also, management of daemon clients also becomes an issue for large build farms (e.g. notification of exceptional events like running out of disk space).

  • Ease of Use
    It is worth restating that ease-of-use is a major factor in practice. Developers will not use QA frameworks unless they add value to the development process. Further, it can be difficult to convince an organization to support the maintenance of automated QA frameworks on a large build farm.
As a final note, the Acro Developer Resources page summarizes the QA tools that the Acro project is using with Hudson to support software development. It is noteworthy that this effort includes QA processes for both C/C++ software and Python software. On another project, we have also used Hudson to summarize tests of Matlab code.

P.S. I want to thank John Siirola for brainstorming about this blog. John has done most of the work setting up the Hudson server that we are using for Acro and related open source software development.

Comments

  1. Geoffrey De SmetJuly 18, 2009 at 12:24 AM

    If you're using Java (or scala or groovy), take a look at maven. I combine these technologies:

    - Maven for the build and the whole POM principle. A pom.xml file states all important project meta-data, such as dependencies, build things.
    - Hudson as the build server. It takes 1 second to hudsonize a maven project: just feed it the pom.xml.
    - Several maven plugins for QA, including:
    -- Dependency report: So if I use org.drools.drools-solver:drools-solver-core:5.0.1 I also get org.codehaus.xstream:stream:1.x in my classpath?
    -- findbugs: So I should use StringBuilder instead of StringBuffer in that peice of code?
    -- pmd and/or checkstyle: So that private field in that class is never read?
    -- surefire reports: So those tests failed?

    Maven has become the defacto standard in Java land (hibernate, spring, ...). It's great: finally I can checkout, build and automatically configure in my IDE (Eclipse, IntelliJ, Netbeans) any open source project and start patching it in under 3 minutes.
    For example. Building drools is as simple as:

    $ svn checkout http://anonsvn.jboss.org/repos/labs/labs/jbossrules/trunk/ drools
    ...
    $ cd drools
    $ mvn -DskipTests clean install
    ...

    ReplyDelete
  2. AnonymousApril 6, 2010 at 1:49 PM

    Any pointers to what else you use for hudson + Python builds?

    Like what you use to change version numbers, upload builds etc..

    ReplyDelete

Post a Comment

Popular posts from this blog

Python Plugin Frameworks

Updated to include pointers to the PyUtilib Component Architecture and PnP. Various Python projects I am working on could benefit from the use of a Plug-in framework.  However, there does not appear to be a standard Python plug-in framework, though there are some mature packages that support plug-ins. Here's a summary of my recent web research: yapsy - This is a simple plug-in framework that was designed specifically to support plug-ins with no external dependencies. Mary Alchin describes a simple plugin framework , with a similar goal.  His classes provide an API for the plugins, with few supporting features (e.g. searching for plugins). André Roberge has a series of posts that describe the application of plugins to refactor a simple calculator application.  The goal of this is to illustrate the requirements for plugin frameworks, with the goal of identifying best practices for plugins. There are some interesting replies to this post, which consider implementati...
Read more

SBOMs for Scientific Software: C++

This is the second of a series of posts exploring the feasibility of generating Software Bills of Materials ( SBOMs ) for complex scientific software. SBOMs provide a list of the components, libraries, and modules that are required to build a piece of software. The United States 2021 Executive Order on Cybersecurity highlights the role of SBOMs to support risk assessments for newly discovered vulnerabilities.  Further, the U.S. National Institute of Standards and Technology (NIST) released its Secure Software Development Framework , which requires SBOM information to be available for software. My previous post  explored the generation of SBOMs for both simple and complex python software packages.  A variety of tools have recently been developed that can create SBOMs for python software, which are relatively easy to install and apply. Standard python installation tools like pip and poetry capture much of the information needed to formulate SBOMs, so developers may not nee...
Read more

Using AsciiDoc for Mathematical Publications

Technical writing is an integral part of my research in computer science and operations research. I have a long history using LaTeX, which is very well suited for writing technical articles that contain mathematical equations as well as code snippets. Although LaTeX can readily generate postscript and PDF output files, I have been unimpressed with tools that generate HTML from LaTeX source. Thus, I was intrigued by AsciiDoc , which promises to generate PDF, HTML and eBook formats. AsciiDoc is used to provide online documentation for software projects, and authors can publish book through O'Rielly using this tool. Thus, this is a well-developed document generation tool. I have successfully prototyped a draft book, Getting Started with Coopr , and you can browse the subversion repository for this document here . Note that the Makefile file specifies build targets for PDF, HTML and eBook files. The advantage of AsciiDoc is that you can use a simple markup language to generate...
Read more