William E. Hart's Blog

This is the second of a series of posts exploring the feasibility of generating Software Bills of Materials ( SBOMs ) for complex scientific software. SBOMs provide a list of the components, libraries, and modules that are required to build a piece of software. The United States 2021 Executive Order on Cybersecurity highlights the role of SBOMs to support risk assessments for newly discovered vulnerabilities.  Further, the U.S. National Institute of Standards and Technology (NIST) released its Secure Software Development Framework , which requires SBOM information to be available for software. My previous post  explored the generation of SBOMs for both simple and complex python software packages.  A variety of tools have recently been developed that can create SBOMs for python software, which are relatively easy to install and apply. Standard python installation tools like pip and poetry capture much of the information needed to formulate SBOMs, so developers may not nee...

This is the first of a series of posts exploring the feasibility of generating Software Bills of Materials ( SBOMs ) for complex scientific software. SBOMs provide a list of the components, libraries, and modules that are required to build a piece of software. The United States 2021 Executive Order on Cybersecurity highlights the role of SBOMs to support risk assessments for newly discovered vulnerabilities.  Further, the U.S. National Institute of Standards and Technology (NIST) released its Secure Software Development Framework , which requires SBOM information to be available for software. Both open source and commercial software are impacted by these policies.  Consequently, developers of scientific software should expect that the use of their software may be restricted in some contexts unless accurate SBOMs can be generated.  The past few years has seen an industry-wide effort to embrace SBOMs and other software security practices highlighted by the U.S. government ...

One of my favorite authors is Vernor Vinge, who explores how the evolution of technology impacts future societies.  Vernor is a former professor of computer science, so it is perhaps no surprise that his tech focus appeals to me.  The novel The Peace War imagines a post-apocalyptic society that is recovering from a world war that was prompted, in part, by a new force field generating device (called the Bobbler). There are all kinds of futuristic technologies described in this book, which is typical for Vinge's literature.  In one scene, the protagonist is about to be ambushed by a group of bad guys. He hands a gun to his teenage companion, who takes out the bad guys without hardly aiming the gun.  The trick is that the gun has computer-guided bullets, which the protagonist directed using his laptop. This sounded like standard sci-fi tech fantasy until I heard about the self-guided bullet that has been developed by Sandia National Laboratories .  Mashable ha...

Software testing is widely recognized as a best practice for software development . Software tests define expected functionality, and they can focus developer efforts by providing an objective assessment of the state of a software project. Additionally, software testing data can provide evidence that a software package can be reliably used. For example, when evaluating whether to try out open source software, I routinely look for software testing data to confirm which platforms the software will run on, the versions of associated software that is used, and test coverage statistics that indicate how much of the the code is tested. Unfortunately, most open source software projects do not publish software test data.  I suspects that this indicates that a small fraction of OSS projects have robust test suites.  However, this also reflects another aspect of the OSS community:  hosting facilities for open source software do not support web-based testing facilities, like Jenk...

This is a blog that I have been meaning to write for some time.   I occasionally take a look at the download statistics for this blog, and recently I was prompted to do this by other bloggers who were reporting their end-of-year statistics (e.g. see Laura McLay’s review of the Punk Rock OR blog ). Unlike Laura, I do not have impressive download statistics to report about the many blogs I have written in 2011; frankly, I did not create many posts. However, an interesting pattern has emerged regarding this blog’s readership:   there are a few key blog posts that are frequently downloaded.   For example, my most frequently downloaded blog post is a survey of Python plugin software, which I wrote in 2009.   I suspect that other bloggers have seen the same thing; they have a few posts that are very popular because people do web searches on that topic.   However, it is worth stepping back and thinking about the implications of this when writing a blog. ...

The Python Optimization Modeling Objects (Pyomo) package is an open source tool for modeling optimization applications in Python. Pyomo can be used to define symbolic problems, create concrete problem instances, and solve these instances with standard solvers. Pyomo provides a capability that is commonly associated with algebraic modeling languages such as AMPL, AIMMS, and GAMS, but Pyomo's modeling objects are embedded within a full-featured high-level programming language with a rich set of supporting libraries. Pyomo leverages the capabilities of the Coopr software library, which integrates Python packages for defining optimizers, modeling optimization applications, and managing computational experiments. Of course, there is very little online documentation describing Pyomo.  However, the first book on Pyomo is set to be published in February! Pyomo - Optimization Modeling in Python . William E. Hart, Carl Laird, Jean-Paul Watson and David L. Woodruff. Springer, 2012. Her...

If you google for "python C++ parser", you will find a variety of internet discussions related to parsing C++ in Python.  C++ cannot be parsed by a LALR parser and it is well-known that parsing C++ is a nontrivial task.  Thus, these discussions generally fall into one of several categories: It is too hard to parse C++ in Python, so use a package like GCC_XML that does this for you.  If you really need to do something in Python, write a wrapper to GCC_XML. It is too hard to perform a complete parse of C++ in Python, but we can use a LALR parser to collect gross structural information from C++ files.  The CppHeaderParser is an example of this type of package, which uses the ply parser to collect information about classes in header files. In the recent release of CxxTest , I included a LALR C++ parser that is similar to CppHeaderParser . CxxTest is a unit testing framework for C++ that is similar in spirit to JUnit, CppUnit, and xUnit. CxxTest is easy to u...

Atjei , the creater of OptimJ, is now closed. The OptimJ product is a Java extension that supports a simplified syntax for specifying optimization models.  It is sad to see this product be abandoned like this.  I do not know of any other optimization modeling tool that directly supports Java. Erwin Kalvelagen noted the demise of Ajeti, and commented that " I believe that some of the more complex data issues in practical modeling are often better (i.e. more efficiently) dealt with in a specialized language than in a traditional programming language." Pyomo is similar to OptimJ, in that it supports optimization modeling in Python. However, a premise that has guided Pyomo development is that users will want to perform modeling in a full-featured programming language. Clearly, traditional programming languages are more verbose and complex than a domain-specific language. However, Python is arguably much simpler to use than Java. In fact, I have gotten feedback from Pyomo ...

Technical writing is an integral part of my research in computer science and operations research. I have a long history using LaTeX, which is very well suited for writing technical articles that contain mathematical equations as well as code snippets. Although LaTeX can readily generate postscript and PDF output files, I have been unimpressed with tools that generate HTML from LaTeX source. Thus, I was intrigued by AsciiDoc , which promises to generate PDF, HTML and eBook formats. AsciiDoc is used to provide online documentation for software projects, and authors can publish book through O'Rielly using this tool. Thus, this is a well-developed document generation tool. I have successfully prototyped a draft book, Getting Started with Coopr , and you can browse the subversion repository for this document here . Note that the Makefile file specifies build targets for PDF, HTML and eBook files. The advantage of AsciiDoc is that you can use a simple markup language to generate...

I stumbled across the following site, which provides download statistics for PyPI Python optimization packages:     http://taichino.appspot.com/pypi_ranking/keyword/optimization It was fun to see the different download package statistics. I'm quite curious that coopr.pysos has the highest number of downloads, since IMHO this package doesn't have much interesting functionality... Hmmm... Another surprise for me is that so many optimization Python packages are _not_ on this list!  I am maintaining the following list of links for Python optimization packages:     https://software.sandia.gov/trac/coopr/wiki/Documentation/RelatedProjects Perhaps these packages were not tagged with the 'optimization' label ... or perhaps they were not downloaded enough to make the list?

I have been maintaining a list of Python optimization packages for a while now on the Coopr Trac pages:  see https://software.sandia.gov/trac/coopr/wiki/Documentation/RelatedProjects .Today, I noticed that if you google for "python optimization packages", this page does not show up right away.  Perhaps Trac pages are index differently?  I'm not sure. Anyway, I thought I'd add this reference here to help others find this list that I'm maintaining...

We are pleased to announce the release of Coopr 3.0 (3.0.4362). Coopr is a collection of Python software packages that supports a diverse set of optimization capabilities for formulating and analyzing optimization models. The following are highlights of this release: - Solvers * More sophisticated logic for solver factory to find ASL and OS solvers * Various solver interface improvements * New Solver results object for efficient representation of variable values * New support for asynchronous progressive hedging - Modeling * Changes in rule semantics to limit rule return values * Changes in the expected order of rule arguments * Constant sums or products can now be used as constraint bounds * Added full support for the !ConstraintList modeling component. - Usability enhancements * More explicit output from runph and runef commands * Added support in runef to write the extensive form in NL format * Add controls for garbage collection in PH - Other * Efficiency impr...

Coopr 2.5(2.5.3890) has just been released ! Coopr is a collection of Python software packages that supports a diverse set of optimization capabilities for formulating and analyzing optimization models. The following are highlights of this release: - Solvers     * MIP solver interface updates to use appropriate objective names     * Added support for suffixes in GUROBI solver interface     * Improved diagnostic analysis of PH solver for the extensive form - Usability enhancements     * Improved robustness of coopr_install     * Fixed Coopr installation problem when using easy_install     * Added a script to launch the CooprAge GUI.     * LP files now are written with the true objective name     * Rework of pyomo command line to create a concise output     * Many efficiency improvements during model generation!     * Many improvements...

Working with and developing Python software is complicated by several factors: Many users do not have the administrative privileges that are needed to install packages in the Python site packages directory, where Python packages are most easily installed Python developers often need to manage many Python projects in the same development environment The Python virtualenv package resolves these issues by allowing users to generate a local Python installation. Thus, a user can create a Python installation into which they can install Python packages using standard tools like easy_install and pip. Unfortunately, getting started with virtualenv is complicated by the fact that you need to install this package to create virtual Python installations. But, if you do not have administrative privileges, then you are stuck. This is the simplest process that I have seen: hg clone http://bitbucket.org/ianb/virtualenv virtualenv/virtualenv.py ENV Unfortunately, the hg command may not be...

I almost gave the following presentation at the INFORMS Annual Meeting: Installing Python Software Packages: The Good, The Bad and the Ugly That is, I was scheduled to give this talk but my session co-organizer ran over and I had to summarize these slides in 5 minutes! Anyway, these slides describe different strategies for installing Python software.  Although I am a big fan of Python software development, robust strategies for software installation remains a challenge.  This talk describes several different installation scenarios: The Good: the user has administrative privileges Installing on Windows with an installer executable Installing with Linux application utility Installing a Python package from the PyPI repository Installing a Python package from source The Bad: the user does not have administrative privileges Using a virtual environment to isolate package installations Using an installer executable on Windows with a virtual environment The Ugly: the ...

The role of the FAST software repository has evolved over the past few years, and it is time to rethink what the goal of FAST is. See Rethinking the goal of FAST for some perspective about FAST. The upshot is that FAST has been reorganized to support an ensemble of independent software packages related to agile software development. One important change in this reorganization has been the decomposition of the FAST Python software into independent components. Some elements of FAST are deprecated, and the rest have been spread out into various packages. See the FAST Blog for further details.

The gcovr command provides a utility for running the gcov command and summarizing code coverage results. This command is inspired by the Python coverage.py package, which provides a similar utility in Python. Further, gcovr can be viewed as a command-line alternative of the lcov utility, which runs gcov and generates an HTML output. Recently, the gcovr script has been broken out into a separate software package, which is managed at https://software.sandia.gov/svn/public/fast/gcovr This package has been uploaded to the Python PyPI repository to facilitate it installation with easy_install: easy_install gcovr Alternatively, the gcovr script can be downloaded directly: https://software.sandia.gov/svn/public/fast/gcovr/trunk/scripts/gcovr See the gcovr Trac page for further details about this tool. Although gcovr is increasingly used to generate coverage statistics within Hudson, the gcovr wiki documents a command-line text summary that I personally find very useful when developing...

A while back I developed EXACT , a Python package for executing computational experiments using an XML-defined process. EXACT was designed to fill a particular niche in software testing: performing computational tests that involve the application of solvers to a suite of test problems. This sort of testing arises a lot when doing functionality testing for scientific software. Unfortunately, EXACT was too complex: The XML specification was complex and difficult to read Experiments with many factors were assigned generic experiment IDs It was hard to replication the execution of specific experiments The experimental results were captured in XML results files that were difficult to browse Even my close collaborators struggled to setup, run and analyze computational experiments! {sigh} I have recently developed the pyutilib.autotest Python package to provide a simpler alternative to EXACT. This package uses a YAML test configuration file to specify the solvers and problems that are ...

The PyUtilib Component Architecture (PCA) is a component architecture in Python that is derived from the Trac component framework . One important extension was to support both singleton and non-singleton plugins. Trac plugins are singletons that are created immediately when the plugin module is loaded (Python is a wonderful language for supporting this type of capability). Singleton plugins are well-suited for Trac, since it is a persistent application, but many other applications need to employ plugins "on demand". Consequently, the PCA supports non-singleton plugins, which need to be explicitly constructed by the end-user. The PCA is widely used in the Coopr project, and most plugins are non-singletons. Until recently, the PCA did not directly support plugin construction on demand. That is, the user needed to know the plugin class name, and plugin construction was done explicitly by the user. However, most plugins in Coopr can be best described as named services . ...

This is just a heads-up that I have setup blogs for the Coopr and PyUtilib software projects: Coopr: https://software.sandia.gov/trac/coopr/blog PyUtilib: https://software.sandia.gov/trac/pyutilib/blog These blogs will document releases and end-user advice and examples. Additionally, there will hopefully be other authors than just me adding content to these blogs. But, I guess we'll see.