William E. Hart's Blog

I almost gave the following presentation at the INFORMS Annual Meeting: Installing Python Software Packages: The Good, The Bad and the Ugly That is, I was scheduled to give this talk but my session co-organizer ran over and I had to summarize these slides in 5 minutes! Anyway, these slides describe different strategies for installing Python software.  Although I am a big fan of Python software development, robust strategies for software installation remains a challenge.  This talk describes several different installation scenarios: The Good: the user has administrative privileges Installing on Windows with an installer executable Installing with Linux application utility Installing a Python package from the PyPI repository Installing a Python package from source The Bad: the user does not have administrative privileges Using a virtual environment to isolate package installations Using an installer executable on Windows with a virtual environment The Ugly: the ...

A while back I developed EXACT , a Python package for executing computational experiments using an XML-defined process. EXACT was designed to fill a particular niche in software testing: performing computational tests that involve the application of solvers to a suite of test problems. This sort of testing arises a lot when doing functionality testing for scientific software. Unfortunately, EXACT was too complex: The XML specification was complex and difficult to read Experiments with many factors were assigned generic experiment IDs It was hard to replication the execution of specific experiments The experimental results were captured in XML results files that were difficult to browse Even my close collaborators struggled to setup, run and analyze computational experiments! {sigh} I have recently developed the pyutilib.autotest Python package to provide a simpler alternative to EXACT. This package uses a YAML test configuration file to specify the solvers and problems that are ...

I earlier mentioned that I developed a plugin framework within my PyUtilib software.  This is now called the PyUtilib Component Architecture (PCA).  The PCA is derived from the Trac component architecture, and it supports advanced features like nonsingleton components, namespaces and caching of component interactions. The PCA includes an independent, self-contained framework core that can be easily integrated into other applications, as well as a variety of extension packages with commonly used components. See The PyUtilib Component Architecture for further details.

Python's setuptools package includes the easy_install script, which provides a convenient mechanism for installing a Python package from the PyPi repository.  Normally, easy_install installs a Python package in the Python site packages directory.  However, I recently discovered that easy_install can download the source for Python package.  For example, the following command downloads the Coopr optimization package into the coopr directory: easy_install -q --editable --build-directory . Coopr I had to browse a variety of web pages before I figured this syntax out.  Enjoy!

After blogging about Python plugin frameworks earlier this year, I wound up implemented a new framework in the PyUtilib software package.  The PyUtilib wiki provides a detailed description of the PyUtilib Plugin Framework , but here's a brief summary: This framework is derived from the Trac plugin framework (a.k.a. component architecture) Provides support for both singleton and non-singleton plugin instances Includes utilities for managing plugins within namespaces The core framework is defined in a single Python module, which can be used independently from PyUtilib PyUtilib also includes commonly use plugins, such as A config-file reader/writer based on ConfigParser Loading utilities for eggs and modules A file manager for temporary files Although I initially resisted the urge to develop my own framework, I was led to develop this because (1) I wanted a light-weight framework like that provided by Trac, but (2) Trac's framework is not particularlly modular within the Trac so...

Updated to include pointers to the PyUtilib Component Architecture and PnP. Various Python projects I am working on could benefit from the use of a Plug-in framework.  However, there does not appear to be a standard Python plug-in framework, though there are some mature packages that support plug-ins. Here's a summary of my recent web research: yapsy - This is a simple plug-in framework that was designed specifically to support plug-ins with no external dependencies. Mary Alchin describes a simple plugin framework , with a similar goal.  His classes provide an API for the plugins, with few supporting features (e.g. searching for plugins). André Roberge has a series of posts that describe the application of plugins to refactor a simple calculator application.  The goal of this is to illustrate the requirements for plugin frameworks, with the goal of identifying best practices for plugins. There are some interesting replies to this post, which consider implementati...

It does not take much to add a lambda function to a Python class. For example, consider the following: >>> class A: pass >>> f = lambda self,x:x >>> setattr(A,"f",f) This code adds the method f to class A . For example: >>> a=A() >>> a.f(1) 1 However, the f method created this way does not have the expected Python name: >>> A.f.__name__ ' ' Further, defining this value is not possible; the instancemethod f does not have a __name__ attribute. The trick is to name the lambda function before defining the class method: >>> class A: pass >>> f = lambda self,x:x >>> f.__name__ = "f" >>> setattr(A,"f",f) >>> f.__name__ 'f' This is simple, but it took too long to figure this out...

There are many applications where you want to apply a code to a variety of data sets, and verify that you get the correct output. In this context, what you want is a test generator, which can dynamically create tests, based on the set of data sets that are available for testing. Unfortunately, this does not appear to be a feature of unittest . The closest I have seen to this, is the support for test generators in the nose package, which extends unittest to provide test discovery mechanisms. However, that test generation feature is somewhat limited; it only applies to test functions that are Python generators, and not to similar class methods. The following example shows how to directly insert new test methods into a unittest.TestCase class : # # A simple example for generating tests in the Python unittest framework # import glob import unittest # # Defining the class that will contain the new tests # class TestCases(unittest.TestCase): pass # # A generic function that performs a te...

In the past year, I have increasingly been using Python to develop a variety of OR-related scientific software. In particular, the Coopr library has been a major focus of this software development. Recently, I have written a paper that will appear in the proceedings of the INFORMS Computing Society Conference 2009 : W. Hart, Python Optimization Modeling Objects (Pyomo) , Proc. INFORMS Computing Society Conference, 2009, (to appear). In this paper, I describe Pyomo, an open-source tool for modeling optimization applications in Python. A key goal of Pyomo is to provide an open-source math programming modeling capability. Although open-source optimization solvers are widely available in packages like COIN-OR , surprisingly few open-source tools have been developed to model optimization applications. Pyomo has been developed in Python because it is a well-used modern programming language that provides a robust foundation for developing and applying scientific software. In this paper, I...