PEP: 405 Title: Python Virtual Environments Version: $Revision$
Last-Modified: $Date$ Author: Carl Meyer <carl@oddbird.net>
BDFL-Delegate: Alyssa Coghlan Status: Final Type: Standards Track Topic:
Packaging Content-Type: text/x-rst Created: 13-Jun-2011 Python-Version:
3.3 Post-History: 24-Oct-2011, 28-Oct-2011, 06-Mar-2012, 24-May-2012
Resolution:
https://mail.python.org/pipermail/python-dev/2012-May/119668.html

Abstract

This PEP proposes to add to Python a mechanism for lightweight "virtual
environments" with their own site directories, optionally isolated from
system site directories. Each virtual environment has its own Python
binary (allowing creation of environments with various Python versions)
and can have its own independent set of installed Python packages in its
site directories, but shares the standard library with the base
installed Python.

Motivation

The utility of Python virtual environments has already been well
established by the popularity of existing third-party
virtual-environment tools, primarily Ian Bicking's virtualenv. Virtual
environments are already widely used for dependency management and
isolation, ease of installing and using Python packages without
system-administrator access, and automated testing of Python software
across multiple Python versions, among other uses.

Existing virtual environment tools suffer from lack of support from the
behavior of Python itself. Tools such as rvirtualenv, which do not copy
the Python binary into the virtual environment, cannot provide reliable
isolation from system site directories. Virtualenv, which does copy the
Python binary, is forced to duplicate much of Python's site module and
manually symlink/copy an ever-changing set of standard-library modules
into the virtual environment in order to perform a delicate
boot-strapping dance at every startup. (Virtualenv must copy the binary
in order to provide isolation, as Python dereferences a symlinked
executable before searching for sys.prefix.)

The PYTHONHOME environment variable, Python's only existing built-in
solution for virtual environments, requires copying/symlinking the
entire standard library into every environment. Copying the whole
standard library is not a lightweight solution, and cross-platform
support for symlinks remains inconsistent (even on Windows platforms
that do support them, creating them often requires administrator
privileges).

A virtual environment mechanism integrated with Python and drawing on
years of experience with existing third-party tools can lower
maintenance, raise reliability, and be more easily available to all
Python users.

Specification

When the Python binary is executed, it attempts to determine its prefix
(which it stores in sys.prefix), which is then used to find the standard
library and other key files, and by the site module to determine the
location of the site-package directories. Currently the prefix is found
(assuming PYTHONHOME is not set) by first walking up the filesystem tree
looking for a marker file (os.py) that signifies the presence of the
standard library, and if none is found, falling back to the build-time
prefix hardcoded in the binary.

This PEP proposes to add a new first step to this search. If a
pyvenv.cfg file is found either adjacent to the Python executable or one
directory above it (if the executable is a symlink, it is not
dereferenced), this file is scanned for lines of the form key = value.
If a home key is found, this signifies that the Python binary belongs to
a virtual environment, and the value of the home key is the directory
containing the Python executable used to create this virtual
environment.

In this case, prefix-finding continues as normal using the value of the
home key as the effective Python binary location, which finds the prefix
of the base installation. sys.base_prefix is set to this value, while
sys.prefix is set to the directory containing pyvenv.cfg.

(If pyvenv.cfg is not found or does not contain the home key,
prefix-finding continues normally, and sys.prefix will be equal to
sys.base_prefix.)

Also, sys.base_exec_prefix is added, and handled similarly with regard
to sys.exec_prefix. (sys.exec_prefix is the equivalent of sys.prefix,
but for platform-specific files; by default it has the same value as
sys.prefix.)

The site and sysconfig standard-library modules are modified such that
the standard library and header files are found relative to
sys.base_prefix / sys.base_exec_prefix, while site-package directories
("purelib" and "platlib", in sysconfig terms) are still found relative
to sys.prefix / sys.exec_prefix.

Thus, a Python virtual environment in its simplest form would consist of
nothing more than a copy or symlink of the Python binary accompanied by
a pyvenv.cfg file and a site-packages directory.

Isolation from system site-packages

By default, a virtual environment is entirely isolated from the
system-level site-packages directories.

If the pyvenv.cfg file also contains a key include-system-site-packages
with a value of true (not case sensitive), the site module will also add
the system site directories to sys.path after the virtual environment
site directories. Thus system-installed packages will still be
importable, but a package of the same name installed in the virtual
environment will take precedence.

PEP 370 user-level site-packages are considered part of the system
site-packages for venv purposes: they are not available from an isolated
venv, but are available from an include-system-site-packages = true
venv.

Creating virtual environments

This PEP also proposes adding a new venv module to the standard library
which implements the creation of virtual environments. This module can
be executed using the -m flag:

    python3 -m venv /path/to/new/virtual/environment

A pyvenv installed script is also provided to make this more convenient:

    pyvenv /path/to/new/virtual/environment

Running this command creates the target directory (creating any parent
directories that don't exist already) and places a pyvenv.cfg file in it
with a home key pointing to the Python installation the command was run
from. It also creates a bin/ (or Scripts on Windows) subdirectory
containing a copy (or symlink) of the python3 executable, and the
pysetup3 script from the packaging standard library module (to
facilitate easy installation of packages from PyPI into the new venv).
And it creates an (initially empty) lib/pythonX.Y/site-packages (or
Lib\site-packages on Windows) subdirectory.

If the target directory already exists an error will be raised, unless
the --clear option was provided, in which case the target directory will
be deleted and virtual environment creation will proceed as usual.

The created pyvenv.cfg file also includes the
include-system-site-packages key, set to true if pyvenv is run with the
--system-site-packages option, false by default.

Multiple paths can be given to pyvenv, in which case an identical venv
will be created, according to the given options, at each provided path.

The venv module also places "shell activation scripts" for POSIX and
Windows systems in the bin or Scripts directory of the venv. These
scripts simply add the virtual environment's bin (or Scripts) directory
to the front of the user's shell PATH. This is not strictly necessary
for use of a virtual environment (as an explicit path to the venv's
python binary or scripts can just as well be used), but it is
convenient.

In order to allow pysetup and other Python package managers to install
packages into the virtual environment the same way they would install
into a normal Python installation, and avoid special-casing virtual
environments in sysconfig beyond using sys.base_prefix in place of
sys.prefix where appropriate, the internal virtual environment layout
mimics the layout of the Python installation itself on each platform. So
a typical virtual environment layout on a POSIX system would be:

    pyvenv.cfg
    bin/python3
    bin/python
    bin/pysetup3
    include/
    lib/python3.3/site-packages/

While on a Windows system:

    pyvenv.cfg
    Scripts/python.exe
    Scripts/python3.dll
    Scripts/pysetup3.exe
    Scripts/pysetup3-script.py
            ... other DLLs and pyds...
    Include/
    Lib/site-packages/

Third-party packages installed into the virtual environment will have
their Python modules placed in the site-packages directory, and their
executables placed in bin/ or Scripts.

Note

On a normal Windows system-level installation, the Python binary itself
wouldn't go inside the "Scripts/" subdirectory, as it does in the
default venv layout. This is useful in a virtual environment so that a
user only has to add a single directory to their shell PATH in order to
effectively "activate" the virtual environment.

Note

On Windows, it is necessary to also copy or symlink DLLs and pyd files
from compiled stdlib modules into the env, because if the venv is
created from a non-system-wide Python installation, Windows won't be
able to find the Python installation's copies of those files when Python
is run from the venv.

Sysconfig install schemes and user-site

This approach explicitly chooses not to introduce a new sysconfig
install scheme for venvs. Rather, by modifying sys.prefix we ensure that
existing install schemes which base locations on sys.prefix will simply
work in a venv. Installation to other install schemes (for instance, the
user-site schemes) whose paths are not relative to sys.prefix, will not
be affected by a venv at all.

It may be feasible to create an alternative implementation of Python
virtual environments based on a virtual-specific sysconfig scheme, but
it would be less robust, as it would require more code to be aware of
whether it is operating within a virtual environment or not.

Copies versus symlinks

The technique in this PEP works equally well in general with a copied or
symlinked Python binary (and other needed DLLs on Windows). Symlinking
is preferable where possible, because in the case of an upgrade to the
underlying Python installation, a Python executable copied in a venv
might become out-of-sync with the installed standard library and require
manual upgrade.

There are some cross-platform difficulties with symlinks:

-   Not all Windows versions support symlinks, and even on those that
    do, creating them often requires administrator privileges.
-   On OS X framework builds of Python, sys.executable is just a stub
    that executes the real Python binary. Symlinking this stub does not
    work; it must be copied. (Fortunately the stub is also small, and
    not changed by bugfix upgrades to Python, so copying it is not an
    issue).

Thus, this PEP proposes to symlink the binary on all platforms except
for Windows, and OS X framework builds. A --symlink option is available
to force the use of symlinks on Windows versions that support them, if
the appropriate permissions are available. (This option has no effect on
OS X framework builds, since symlinking can never work there, and has no
advantages).

On Windows, if --symlink is not used, this means that if the underlying
Python installation is upgraded, the Python binary and DLLs in the venv
should be updated, or there could be issues of mismatch with the
upgraded standard library. The pyvenv script accepts a --upgrade option
for easily performing this upgrade on an existing venv.

Include files

Current virtualenv handles include files in this way:

On POSIX systems where the installed Python's include files are found in
${base_prefix}/include/pythonX.X, virtualenv creates ${venv}/include/
and symlinks ${base_prefix}/include/pythonX.X to
${venv}/include/pythonX.X. On Windows, where Python's include files are
found in {{ sys.prefix }}/Include and symlinks are not reliably
available, virtualenv copies {{ sys.prefix }}/Include to
${venv}/Include. This ensures that extension modules built and installed
within the virtualenv will always find the Python header files they need
in the expected location relative to sys.prefix.

This solution is not ideal when an extension module installs its own
header files, as the default installation location for those header
files may be a symlink to a system directory that may not be writable.
One installer, pip, explicitly works around this by installing header
files to a nonstandard location ${venv}/include/site/pythonX.X/, as in
Python there's currently no standard abstraction for a site-specific
include directory.

This PEP proposes a slightly different approach, though one with
essentially the same effect and the same set of advantages and
disadvantages. Rather than symlinking or copying include files into the
venv, we simply modify the sysconfig schemes so that header files are
always sought relative to base_prefix rather than prefix. (We also
create an include/ directory within the venv, so installers have
somewhere to put include files installed within the env).

Better handling of include files in distutils/packaging and, by
extension, pyvenv, is an area that may deserve its own future PEP. For
now, we propose that the behavior of virtualenv has thus far proved
itself to be at least "good enough" in practice.

API

The high-level method described above makes use of a simple API which
provides mechanisms for third-party virtual environment creators to
customize environment creation according to their needs.

The venv module contains an EnvBuilder class which accepts the following
keyword arguments on instantiation:

-   system_site_packages - A Boolean value indicating that the system
    Python site-packages should be available to the environment.
    Defaults to False.
-   clear - A Boolean value which, if true, will delete any existing
    target directory instead of raising an exception. Defaults to False.
-   symlinks - A Boolean value indicating whether to attempt to symlink
    the Python binary (and any necessary DLLs or other binaries, e.g.
    pythonw.exe), rather than copying. Defaults to False.

The instantiated env-builder has a create method, which takes as
required argument the path (absolute or relative to the current
directory) of the target directory which is to contain the virtual
environment. The create method either creates the environment in the
specified directory, or raises an appropriate exception.

The venv module also provides a module-level create function as a
convenience:

    def create(env_dir,
               system_site_packages=False, clear=False, use_symlinks=False):
        builder = EnvBuilder(
            system_site_packages=system_site_packages,
            clear=clear,
            use_symlinks=use_symlinks)
        builder.create(env_dir)

Creators of third-party virtual environment tools are free to use the
provided EnvBuilder class as a base class.

The create method of the EnvBuilder class illustrates the hooks
available for customization:

    def create(self, env_dir):
        """
        Create a virtualized Python environment in a directory.

        :param env_dir: The target directory to create an environment in.

        """
        env_dir = os.path.abspath(env_dir)
        context = self.create_directories(env_dir)
        self.create_configuration(context)
        self.setup_python(context)
        self.post_setup(context)

Each of the methods create_directories, create_configuration,
setup_python, and post_setup can be overridden. The functions of these
methods are:

-   create_directories - creates the environment directory and all
    necessary directories, and returns a context object. This is just a
    holder for attributes (such as paths), for use by the other methods.
-   create_configuration - creates the pyvenv.cfg configuration file in
    the environment.
-   setup_python - creates a copy of the Python executable (and, under
    Windows, DLLs) in the environment.
-   post_setup - A (no-op by default) hook method which can be
    overridden in third party subclasses to pre-install packages or
    install scripts in the virtual environment.

In addition, EnvBuilder provides a utility method that can be called
from post_setup in subclasses to assist in installing custom scripts
into the virtual environment. The method install_scripts accepts as
arguments the context object (see above) and a path to a directory. The
directory should contain subdirectories "common", "posix", "nt", each
containing scripts destined for the bin directory in the environment.
The contents of "common" and the directory corresponding to os.name are
copied after doing some text replacement of placeholders:

-   __VENV_DIR__ is replaced with absolute path of the environment
    directory.
-   __VENV_NAME__ is replaced with the environment name (final path
    segment of environment directory).
-   __VENV_BIN_NAME__ is replaced with the name of the bin directory
    (either bin or Scripts).
-   __VENV_PYTHON__ is replaced with the absolute path of the
    environment's executable.

The DistributeEnvBuilder subclass in the reference implementation
illustrates how the customization hook can be used in practice to
pre-install Distribute into the virtual environment. It's not envisaged
that DistributeEnvBuilder will be actually added to Python core, but it
makes the reference implementation more immediately useful for testing
and exploratory purposes.

Backwards Compatibility

Splitting the meanings of sys.prefix

Any virtual environment tool along these lines (which attempts to
isolate site-packages, while still making use of the base Python's
standard library with no need for it to be symlinked into the virtual
environment) is proposing a split between two different meanings (among
others) that are currently both wrapped up in sys.prefix: the answers to
the questions "Where is the standard library?" and "Where is the
site-packages location where third-party modules should be installed?"

This split could be handled by introducing a new sys attribute for
either the former prefix or the latter prefix. Either option potentially
introduces some backwards-incompatibility with software written to
assume the other meaning for sys.prefix. (Such software should
preferably be using the APIs in the site and sysconfig modules to answer
these questions rather than using sys.prefix directly, in which case
there is no backwards-compatibility issue, but in practice sys.prefix is
sometimes used.)

The documentation for sys.prefix describes it as "A string giving the
site-specific directory prefix where the platform independent Python
files are installed," and specifically mentions the standard library and
header files as found under sys.prefix. It does not mention
site-packages.

Maintaining this documented definition would mean leaving sys.prefix
pointing to the base system installation (which is where the standard
library and header files are found), and introducing a new value in sys
(something like sys.site_prefix) to point to the prefix for
site-packages. This would maintain the documented semantics of
sys.prefix, but risk breaking isolation if third-party code uses
sys.prefix rather than sys.site_prefix or the appropriate site API to
find site-packages directories.

The most notable case is probably setuptools and its fork distribute,
which mostly use distutils and sysconfig APIs, but do use sys.prefix
directly to build up a list of site directories for pre-flight checking
where pth files can usefully be placed.

Otherwise, a Google Code Search turns up what appears to be a roughly
even mix of usage between packages using sys.prefix to build up a
site-packages path and packages using it to e.g. eliminate the
standard-library from code-execution tracing.

Although it requires modifying the documented definition of sys.prefix,
this PEP prefers to have sys.prefix point to the virtual environment
(where site-packages is found), and introduce sys.base_prefix to point
to the standard library and Python header files. Rationale for this
choice:

-   It is preferable to err on the side of greater isolation of the
    virtual environment.
-   Virtualenv already modifies sys.prefix to point at the virtual
    environment, and in practice this has not been a problem.
-   No modification is required to setuptools/distribute.

Impact on other Python implementations

The majority of this PEP's changes occur in the standard library, which
is shared by other Python implementations and should not present any
problem.

Other Python implementations will need to replicate the new
sys.prefix-finding behavior of the interpreter bootstrap, including
locating and parsing the pyvenv.cfg file, if it is present.

Reference Implementation

The reference implementation is found in a clone of the CPython
Mercurial repository. To test it, build and run
bin/pyvenv /path/to/new/venv to create a virtual environment.

Copyright

This document has been placed in the public domain.



  Local Variables: mode: indented-text indent-tabs-mode: nil
  sentence-end-double-space: t fill-column: 70 coding: utf-8 End: