William E. Hart's Blog

I almost gave the following presentation at the INFORMS Annual Meeting: Installing Python Software Packages: The Good, The Bad and the Ugly That is, I was scheduled to give this talk but my session co-organizer ran over and I had to summarize these slides in 5 minutes! Anyway, these slides describe different strategies for installing Python software.  Although I am a big fan of Python software development, robust strategies for software installation remains a challenge.  This talk describes several different installation scenarios: The Good: the user has administrative privileges Installing on Windows with an installer executable Installing with Linux application utility Installing a Python package from the PyPI repository Installing a Python package from source The Bad: the user does not have administrative privileges Using a virtual environment to isolate package installations Using an installer executable on Windows with a virtual environment The Ugly: the ...

The role of the FAST software repository has evolved over the past few years, and it is time to rethink what the goal of FAST is. See Rethinking the goal of FAST for some perspective about FAST. The upshot is that FAST has been reorganized to support an ensemble of independent software packages related to agile software development. One important change in this reorganization has been the decomposition of the FAST Python software into independent components. Some elements of FAST are deprecated, and the rest have been spread out into various packages. See the FAST Blog for further details.

The gcovr command provides a utility for running the gcov command and summarizing code coverage results. This command is inspired by the Python coverage.py package, which provides a similar utility in Python. Further, gcovr can be viewed as a command-line alternative of the lcov utility, which runs gcov and generates an HTML output. Recently, the gcovr script has been broken out into a separate software package, which is managed at https://software.sandia.gov/svn/public/fast/gcovr This package has been uploaded to the Python PyPI repository to facilitate it installation with easy_install: easy_install gcovr Alternatively, the gcovr script can be downloaded directly: https://software.sandia.gov/svn/public/fast/gcovr/trunk/scripts/gcovr See the gcovr Trac page for further details about this tool. Although gcovr is increasingly used to generate coverage statistics within Hudson, the gcovr wiki documents a command-line text summary that I personally find very useful when developing...

I earlier mentioned that I developed a plugin framework within my PyUtilib software.  This is now called the PyUtilib Component Architecture (PCA).  The PCA is derived from the Trac component architecture, and it supports advanced features like nonsingleton components, namespaces and caching of component interactions. The PCA includes an independent, self-contained framework core that can be easily integrated into other applications, as well as a variety of extension packages with commonly used components. See The PyUtilib Component Architecture for further details.

Coopr is a collection of Python optimization-related packages that supports a diverse set of optimization capabilities for formulating and analyzing optimization models. The following are key Coopr capabilities that are driving Coopr development: Pyomo: Formulate algebraic models within Python's modern programming language PySP: Generic solvers for stochastic programming problems COLIN: Scripts that simplify IO between optimizers and black-box applications SUCASA: Customize MIP solvers to expose model structure to the MIP solver engine See https://software.sandia.gov/trac/coopr/wiki/GettingStarted for instructions for getting started with Coopr. An installation script, coopr_install, is provided to simplify the installation of Coopr packages along with the third-party Python packages that they depend on. This installer can also automatically install extension packages from Coin Bazaar. See http://groups.google.com/group/coopr-forum/topics for online discussions of Coopr. Two ...

After blogging about Python plugin frameworks earlier this year, I wound up implemented a new framework in the PyUtilib software package.  The PyUtilib wiki provides a detailed description of the PyUtilib Plugin Framework , but here's a brief summary: This framework is derived from the Trac plugin framework (a.k.a. component architecture) Provides support for both singleton and non-singleton plugin instances Includes utilities for managing plugins within namespaces The core framework is defined in a single Python module, which can be used independently from PyUtilib PyUtilib also includes commonly use plugins, such as A config-file reader/writer based on ConfigParser Loading utilities for eggs and modules A file manager for temporary files Although I initially resisted the urge to develop my own framework, I was led to develop this because (1) I wanted a light-weight framework like that provided by Trac, but (2) Trac's framework is not particularlly modular within the Trac so...

Updated to include pointers to the PyUtilib Component Architecture and PnP. Various Python projects I am working on could benefit from the use of a Plug-in framework.  However, there does not appear to be a standard Python plug-in framework, though there are some mature packages that support plug-ins. Here's a summary of my recent web research: yapsy - This is a simple plug-in framework that was designed specifically to support plug-ins with no external dependencies. Mary Alchin describes a simple plugin framework , with a similar goal.  His classes provide an API for the plugins, with few supporting features (e.g. searching for plugins). André Roberge has a series of posts that describe the application of plugins to refactor a simple calculator application.  The goal of this is to illustrate the requirements for plugin frameworks, with the goal of identifying best practices for plugins. There are some interesting replies to this post, which consider implementati...

It does not take much to add a lambda function to a Python class. For example, consider the following: >>> class A: pass >>> f = lambda self,x:x >>> setattr(A,"f",f) This code adds the method f to class A . For example: >>> a=A() >>> a.f(1) 1 However, the f method created this way does not have the expected Python name: >>> A.f.__name__ ' ' Further, defining this value is not possible; the instancemethod f does not have a __name__ attribute. The trick is to name the lambda function before defining the class method: >>> class A: pass >>> f = lambda self,x:x >>> f.__name__ = "f" >>> setattr(A,"f",f) >>> f.__name__ 'f' This is simple, but it took too long to figure this out...

Consider the following use of the standard Unix time command: /usr/bin/time ls -R / If the SIGTERM signal is sent to the time process, then the ls process will continue! This is an unexpected behavior, which is not well-documented. Normally, this is not much of an issue; the process that is monitored will simply terminate quietly. However, when the time utility is used in interactive applications, process interrupts can lead to many unexpected rogue processes. The timer command is a modification of the UNIX timing utility that behaves as expected. When using timer , the SIGTERM signal is sent to the process, which terminates it as expected. The timer command is available in the UTILIB software library, but it can be compiled independently.

There are many tools available that can be used to monitor memory usage in computer programs. However, there are few tools that can be applied to monitor the memory usage of a specific process in an automated manner. Most memory monitoring tools provide a gui that a user can monitor. However, these tools are not useful in contexts where memory must be monitored repeatedly. For example, automated software tests may require checks to validate that the memory usage does not exceed expected limits. The memmon command is a new memory monitoring tool that is included in the UTILIB software library. memmon provides a convenient mechanism to report the maximum amount of memory that a process uses. The memmon command requires the absolute path to the command that will be executed. Beyond that, its default syntax is quite simple: $ ./memmon /bin/sleep 1 53768 Kb used The memmon command can also be used to terminate a process whose memory exceeds a specified threshold: $ ./memmon -k 10 /bin/...

I have not blogged much this fall because I have been busy managing a variety of software releases.  I plan to include further details in upcoming blogs, but I thought I would summarize these releases here: acro 2.0 - Acro is A Common Repository for Optimizers that integrates a rich variety of optimization libraries and solvers that have been developed for large-scale engineering and scientific applications. Acro was developed to facilitate the design, development, integration and support of optimization software libraries. Thus, Acro includes both individual optimization solvers as well as optimization frameworks that provide abstract interfaces for flexible interoperability of solver components. Furthermore, many solvers included in Acro can exploit parallel computing resources to solve optimization problems more quickly. utilib 4.0 - Utilib is a library of general-purpose C++ utilities, similar in spirit to the Boost libraries. While generally treated as an Acro package, Utilib i...

I have recently joined the editorial board of the new journal Mathematical Programming Computation , which publishes original research articles that are at the intersection of math programming and computing. This journal reflects the growing role of computation in operations research, where real-world applications often require the application of complex software packages to analyze mathematical models. This journal will include articles that report on innovative software, comparative tests, modeling environments, libraries of data, and/or applications. A main feature of the journal is the inclusion of accompanying software and data with submitted manuscripts. The journal's review process includes the evaluation and testing of the accompanying software. Where possible, the review will aim for verification of reported computational results. Topics covered in Mathematical Programming Computation include linear programming, convex optimization, nonlinear optimization, stochastic opti...

By necessity, I have become quite adept at digging through webspace with search engines like google to figure out "how to do X". But occasionally, it is difficult to get a sense of whether something is easy from written instructions. For example, I recently tried to install PyQT, a Python interface to the popular QT application interface library, and here's the error that I got when trying to use nmake to build the SIP library (which PyQT uses): C:\Python25\sip-4.7.6\sip-4.7.6\siplib>nmake Microsoft (R) Program Maintenance Utility Version 8.00.50727.762 Copyright (C) Microsoft Corporation. All rights reserved. cl -c -nologo -Zm200 -O2 -MD -W0 -DUNICODE -DWIN32 -DQT_LARGEFILE_SUPPORT -I. -IC:\Python25\include -Fo @C:\DOCUME~1\wehart\LOCALS~1\Temp\nm271.tmp NMAKE : fatal error U1077: '"C:\Program Files\Microsoft Visual Studio 8\VC\bin\cl.EXE"' : return code '0xc0000135' Stop. I had more than a little difficulty figuring out what the ret...