Testing Open Source Software

Software testing is widely recognized as a best practice for software development. Software tests define expected functionality, and they can focus developer efforts by providing an objective assessment of the state of a software project. Additionally, software testing data can provide evidence that a software package can be reliably used. For example, when evaluating whether to try out open source software, I routinely look for software testing data to confirm which platforms the software will run on, the versions of associated software that is used, and test coverage statistics that indicate how much of the the code is tested.

Unfortunately, most open source software projects do not publish software test data.  I suspects that this indicates that a small fraction of OSS projects have robust test suites.  However, this also reflects another aspect of the OSS community:  hosting facilities for open source software do not support web-based testing facilities, like Jenkins, that can be used by developers to remotely launch jobs on test machines with a variety of different configurations. This is not totally unexpected, since testing can be computationally intensive.

Recently, I learned about CloudBees, which provides cloud services for building, running and managing Java applications. Happily CloudBees makes its Dev@cloud service freely available to open source projects! This includes the Jenkins testing service, which provides a limited number of CPU hours each month for testing an OSS project.  For example, the CxxTest project now hosts tests on a CloudBees Jenkins server.  Cool!

Comments

Post a Comment

Popular posts from this blog

Python Plugin Frameworks

Updated to include pointers to the PyUtilib Component Architecture and PnP. Various Python projects I am working on could benefit from the use of a Plug-in framework.  However, there does not appear to be a standard Python plug-in framework, though there are some mature packages that support plug-ins. Here's a summary of my recent web research: yapsy - This is a simple plug-in framework that was designed specifically to support plug-ins with no external dependencies. Mary Alchin describes a simple plugin framework , with a similar goal.  His classes provide an API for the plugins, with few supporting features (e.g. searching for plugins). André Roberge has a series of posts that describe the application of plugins to refactor a simple calculator application.  The goal of this is to illustrate the requirements for plugin frameworks, with the goal of identifying best practices for plugins. There are some interesting replies to this post, which consider implementati...
Read more

SBOMs for Scientific Software: C++

This is the second of a series of posts exploring the feasibility of generating Software Bills of Materials ( SBOMs ) for complex scientific software. SBOMs provide a list of the components, libraries, and modules that are required to build a piece of software. The United States 2021 Executive Order on Cybersecurity highlights the role of SBOMs to support risk assessments for newly discovered vulnerabilities.  Further, the U.S. National Institute of Standards and Technology (NIST) released its Secure Software Development Framework , which requires SBOM information to be available for software. My previous post  explored the generation of SBOMs for both simple and complex python software packages.  A variety of tools have recently been developed that can create SBOMs for python software, which are relatively easy to install and apply. Standard python installation tools like pip and poetry capture much of the information needed to formulate SBOMs, so developers may not nee...
Read more

Using AsciiDoc for Mathematical Publications

Technical writing is an integral part of my research in computer science and operations research. I have a long history using LaTeX, which is very well suited for writing technical articles that contain mathematical equations as well as code snippets. Although LaTeX can readily generate postscript and PDF output files, I have been unimpressed with tools that generate HTML from LaTeX source. Thus, I was intrigued by AsciiDoc , which promises to generate PDF, HTML and eBook formats. AsciiDoc is used to provide online documentation for software projects, and authors can publish book through O'Rielly using this tool. Thus, this is a well-developed document generation tool. I have successfully prototyped a draft book, Getting Started with Coopr , and you can browse the subversion repository for this document here . Note that the Makefile file specifies build targets for PDF, HTML and eBook files. The advantage of AsciiDoc is that you can use a simple markup language to generate...
Read more