Friday, November 26, 2010

Getting Started with a Virtual Python Installation

Working with and developing Python software is complicated by several factors:
  • Many users do not have the administrative privileges that are needed to install packages in the Python site packages directory, where Python packages are most easily installed
  • Python developers often need to manage many Python projects in the same development environment
The Python virtualenv package resolves these issues by allowing users to generate a local Python installation. Thus, a user can create a Python installation into which they can install Python packages using standard tools like easy_install and pip.

Unfortunately, getting started with virtualenv is complicated by the fact that you need to install this package to create virtual Python installations. But, if you do not have administrative privileges, then you are stuck. This is the simplest process that I have seen:
hg clone
virtualenv/ ENV
Unfortunately, the hg command may not be preinstalled on your machine. It is not available on any of mine...

The pyutilib.virtualenv package contains the vpy_install script, which can bootstrap the setup of virtualenv. This script extends the functionality of the virtualenv driver script to allow direct, and it customization of this initial environment. The vpy_install script can be easily downloaded from the PyUtilib wiki, or it can be downloaded with commands like wget or curl:
vpy_install ENV

Wednesday, November 10, 2010

Installing Python Software Packages: The Good, The Bad and the Ugly

I almost gave the following presentation at the INFORMS Annual Meeting:

That is, I was scheduled to give this talk but my session co-organizer ran over and I had to summarize these slides in 5 minutes!

Anyway, these slides describe different strategies for installing Python software.  Although I am a big fan of Python software development, robust strategies for software installation remains a challenge.  This talk describes several different installation scenarios:

The Good: the user has administrative privileges
  1. Installing on Windows with an installer executable
  2. Installing with Linux application utility
  3. Installing a Python package from the PyPI repository
  4. Installing a Python package from source

The Bad: the user does not have administrative privileges
  1. Using a virtual environment to isolate package installations
  2. Using an installer executable on Windows with a virtual environment

    The Ugly: the user needs to install an extension package from source
    1. Installing a Python extension package from source
    2. PyCoinInstall – Managing builds for Python extension packages
    The last item referring to PyCoinInstall describes a utility being developed for the COIN-OR software, which is used within the operations research community.  COIN-OR includes a variety of Python and C++ software packages, and this script uses a simple plug-in system to support the management of package builds and installation.  Of course, this mostly does the easy packages right now...

    While preparing these slides I realized that the vpy_install script I use for setting up virtual environments is probably of broader interest.  I'll say more about this in a little bit.  In the meantime, checkout the slides!

    Tuesday, August 31, 2010

    Reworking FAST

    The role of the FAST software repository has evolved over the past few years, and it is time to rethink what the goal of FAST is. See Rethinking the goal of FAST for some perspective about FAST. The upshot is that FAST has been reorganized to support an ensemble of independent software packages related to agile software development.

    One important change in this reorganization has been the decomposition of the FAST Python software into independent components. Some elements of FAST are deprecated, and the rest have been spread out into various packages. See the FAST Blog for further details.

    Sunday, August 29, 2010

    Update for gcovr

    The gcovr command provides a utility for running the gcov command and summarizing code coverage results. This command is inspired by the Python package, which provides a similar utility in Python. Further, gcovr can be viewed as a command-line alternative of the lcov utility, which runs gcov and generates an HTML output.

    Recently, the gcovr script has been broken out into a separate software package, which is managed at

    This package has been uploaded to the Python PyPI repository to facilitate it installation with easy_install:

    easy_install gcovr

    Alternatively, the gcovr script can be downloaded directly:

    See the gcovr Trac page for further details about this tool. Although gcovr is increasingly used to generate coverage statistics within Hudson, the gcovr wiki documents a command-line text summary that I personally find very useful when developing tests to improve code coverage.

    Monday, June 28, 2010

    A New Python Package: pyutilib.autotest

    A while back I developed EXACT, a Python package for executing computational experiments using an XML-defined process. EXACT was designed to fill a particular niche in software testing: performing computational tests that involve the application of solvers to a suite of test problems. This sort of testing arises a lot when doing functionality testing for scientific software.

    Unfortunately, EXACT was too complex:
    • The XML specification was complex and difficult to read
    • Experiments with many factors were assigned generic experiment IDs
    • It was hard to replication the execution of specific experiments
    • The experimental results were captured in XML results files that were difficult to browse
    Even my close collaborators struggled to setup, run and analyze computational experiments! {sigh}

    I have recently developed the pyutilib.autotest Python package to provide a simpler alternative to EXACT. This package uses a YAML test configuration file to specify the solvers and problems that are exercised in a test. The tests setup and executed with Python's unittest package, which allows the user to apply a rich set of testing methods. The user can specify a test driver in Python that defines how each test is executed, using the solver and problem options that are specified in the configuration file.

    Here's a simple example of a YAML test specification, where the goal is to apply the UNIX cat command with different options to different test files:
    driver: example

    name: cat
    cat_options: -n

    file: file1.txt
    file: file2.txt

    See the pyutilib.autotest PyPI documentation for further details.

    Monday, June 21, 2010

    Dynamic Service Creation in the PyUtilib Component Architecture

    The PyUtilib Component Architecture (PCA) is a component architecture in Python that is derived from the Trac component framework. One important extension was to support both singleton and non-singleton plugins. Trac plugins are singletons that are created immediately when the plugin module is loaded (Python is a wonderful language for supporting this type of capability). Singleton plugins are well-suited for Trac, since it is a persistent application, but many other applications need to employ plugins "on demand". Consequently, the PCA supports non-singleton plugins, which need to be explicitly constructed by the end-user.

    The PCA is widely used in the Coopr project, and most plugins are non-singletons. Until recently, the PCA did not directly support plugin construction on demand. That is, the user needed to know the plugin class name, and plugin construction was done explicitly by the user. However, most plugins in Coopr can be best described as named services. For example, Coopr optimizers are implemented as plugins and each optimizer has a unique name.

    The PCA has recently been extended to support the alias function, which can be used to declare the name of a plugin with respect to a particular interface. For example, consider the following definitions of the MyPlugin and YourPlugin plugin classes:

    class ITask(Interface):

    def print(self):

    class MyPlugin(Plugin):

    alias('my', ITask)

    def print(self):
    print "My Plugin"

    class YourPlugin(Plugin):

    alias('your', ITask)

    def print(self):
    print "Your Plugin"

    These plugins are non-singletons, so they are only registered and used if they are constructed. The alias function allows these plugins to be constructed with a factory class that is supported by the PCA. For example:

    factory = CreatePluginFactory(ITask)

    p1 = factory('my')
    p2 = factory('your')

    The CreatePluginFactory class is provided by the PCA, and it provides a functor that manages the creation of plugins whose names are registered with the alias function.

    Blogs for Coopr and PyUtilib

    This is just a heads-up that I have setup blogs for the Coopr and PyUtilib software projects:
    These blogs will document releases and end-user advice and examples. Additionally, there will hopefully be other authors than just me adding content to these blogs. But, I guess we'll see.

    Friday, April 16, 2010

    PyUtilib Component Architecture

    I earlier mentioned that I developed a plugin framework within my PyUtilib software.  This is now called the PyUtilib Component Architecture (PCA).  The PCA is derived from the Trac component architecture, and it supports advanced features like nonsingleton components, namespaces and caching of component interactions. The PCA includes an independent, self-contained framework core that can be easily integrated into other applications, as well as a variety of extension packages with commonly used components.

    See The PyUtilib Component Architecture for further details.

    Recent Coopr Developments

    Coopr is a collection of Python optimization-related packages that supports a diverse set of optimization capabilities for formulating and analyzing optimization models. The following are key Coopr capabilities that are driving Coopr development:

    • Pyomo: Formulate algebraic models within Python's modern programming language
    • PySP: Generic solvers for stochastic programming problems
    • COLIN: Scripts that simplify IO between optimizers and black-box applications
    • SUCASA: Customize MIP solvers to expose model structure to the MIP solver engine

    See for instructions for getting started with Coopr. An installation script, coopr_install, is provided to simplify the installation of Coopr packages along with the third-party Python packages that they depend on. This installer can also automatically install extension packages from Coin Bazaar. See for online discussions of Coopr.

    Two recent Coopr developments are noteworthy:

    I. Coopr development is now co-hosted by Sandia National Laboratories and COIN-OR:

    Coopr includes wrappers for COIN-OR solvers, and Coopr directly leverages the COIN-OR Coin
    Bazaar project to host plug-in packages for Coopr (e.g., see the coopr.neos package in Coin Bazaar).

    II. We have recently released Coopr 2.3. This release includes
    • A preliminary Gurobi solver interface
    • Extended syntax for data command files:
      o 'include' command to load other data command files
      o 'import' command to load data from other sources
      o namespaces to next data declarations
    • The coopr_install script can install packages from Coin Bazaar
    • New conversion scripts to generate LP or NL files from Pyomo models
    • Solvers now extract standard suffix information
    • Various fixes to PySP solvers
    • Full support for PyPI. You can install from PyPI using the 'Coopr' package name.