PEP: 411 Title: Provisional packages in the Python standard library
Version: $Revision$ Last-Modified: $Date$ Author: Alyssa Coghlan
<ncoghlan@gmail.com>, Eli Bendersky <eliben@gmail.com> Status:
Superseded Type: Informational Content-Type: text/x-rst Created:
10-Feb-2012 Python-Version: 3.3 Post-History: 10-Feb-2012, 24-Mar-2012

Note

This PEP has been marked as Superseded. A decade after this PEP was
written, experience has shown this is a rarely used feature in managing
the standard library. It has also not helped prevent people from relying
too heavily on provisional modules, such that changes can still cause
significant breakage in the community.

Abstract

The process of including a new package into the Python standard library
is hindered by the API lock-in and promise of backward compatibility
implied by a package being formally part of Python. This PEP describes a
methodology for marking a standard library package "provisional" for the
period of a single feature release. A provisional package may have its
API modified prior to "graduating" into a "stable" state. On one hand,
this state provides the package with the benefits of being formally part
of the Python distribution. On the other hand, the core development team
explicitly states that no promises are made with regards to the
stability of the package's API, which may change for the next release.
While it is considered an unlikely outcome, such packages may even be
removed from the standard library without a deprecation period if the
concerns regarding their API or maintenance prove well-founded.

Proposal - a documented provisional state

Whenever the Python core development team decides that a new package
should be included into the standard library, but isn't entirely sure
about whether the package's API is optimal, the package can be included
and marked as "provisional".

In the next feature release, the package may either be "graduated" into
a normal "stable" state in the standard library, remain in provisional
state, or be rejected and removed entirely from the Python source tree.
If the package ends up graduating into the stable state after being
provisional, its API may be changed according to accumulated feedback.
The core development team explicitly makes no guarantees about API
stability and backward compatibility of provisional packages.

Marking a package provisional

A package will be marked provisional by a notice in its documentation
page and its docstring. The following paragraph will be added as a note
at the top of the documentation page:

  The <X> package has been included in the standard library on a
  provisional basis. Backwards incompatible changes (up to and including
  removal of the package) may occur if deemed necessary by the core
  developers.

The phrase "provisional basis" will then be a link to the glossary term
"provisional package", defined as:

  A provisional package is one which has been deliberately excluded from
  the standard library's backwards compatibility guarantees. While major
  changes to such packages are not expected, as long as they are marked
  provisional, backwards incompatible changes (up to and including
  removal of the package) may occur if deemed necessary by core
  developers. Such changes will not be made gratuitously -- they will
  occur only if serious flaws are uncovered that were missed prior to
  the inclusion of the package.

  This process allows the standard library to continue to evolve over
  time, without locking in problematic design errors for extended
  periods of time. See PEP 411 for more details.

The following will be added to the start of the package's docstring:

  The API of this package is currently provisional. Refer to the
  documentation for details.

Moving a package from the provisional to the stable state simply implies
removing these notes from its documentation page and docstring.

Which packages should go through the provisional state

We expect most packages proposed for addition into the Python standard
library to go through a feature release in the provisional state. There
may, however, be some exceptions, such as packages that use a
pre-defined API (for example lzma, which generally follows the API of
the existing bz2 package), or packages with an API that has wide
acceptance in the Python development community.

In any case, packages that are proposed to be added to the standard
library, whether via the provisional state or directly, must fulfill the
acceptance conditions set by PEP 2.

Criteria for "graduation"

In principle, most provisional packages should eventually graduate to
the stable standard library. Some reasons for not graduating are:

-   The package may prove to be unstable or fragile, without sufficient
    developer support to maintain it.
-   A much better alternative package may be found during the preview
    release.

Essentially, the decision will be made by the core developers on a
per-case basis. The point to emphasize here is that a package's
inclusion in the standard library as "provisional" in some release does
not guarantee it will continue being part of Python in the next release.
At the same time, the bar for making changes in a provisional package is
quite high. We expect that most of the API of most provisional packages
will be unchanged at graduation. Withdrawals are expected to be rare.

Rationale

Benefits for the core development team

Currently, the core developers are really reluctant to add new
interfaces to the standard library. This is because as soon as they're
published in a release, API design mistakes get locked in due to
backward compatibility concerns.

By gating all major API additions through some kind of a provisional
mechanism for a full release, we get one full release cycle of community
feedback before we lock in the APIs with our standard backward
compatibility guarantee.

We can also start integrating provisional packages with the rest of the
standard library early, so long as we make it clear to packagers that
the provisional packages should not be considered optional. The only
difference between provisional APIs and the rest of the standard library
is that provisional APIs are explicitly exempted from the usual backward
compatibility guarantees.

Benefits for end users

For future end users, the broadest benefit lies in a better
"out-of-the-box" experience - rather than being told "oh, the standard
library tools for task X are horrible, download this 3rd party library
instead", those superior tools are more likely to be just be an import
away.

For environments where developers are required to conduct due diligence
on their upstream dependencies (severely harming the cost-effectiveness
of, or even ruling out entirely, much of the material on PyPI), the key
benefit lies in ensuring that all packages in the provisional state are
clearly under python-dev's aegis from at least the following
perspectives:

-   Licensing: Redistributed by the PSF under a Contributor Licensing
    Agreement.
-   Documentation: The documentation of the package is published and
    organized via the standard Python documentation tools (i.e. ReST
    source, output generated with Sphinx and published on
    http://docs.python.org).
-   Testing: The package test suites are run on the python.org buildbot
    fleet and results published via http://www.python.org/dev/buildbot.
-   Issue management: Bugs and feature requests are handled on
    http://bugs.python.org
-   Source control: The master repository for the software is published
    on http://hg.python.org.

Candidates for provisional inclusion into the standard library

For Python 3.3, there are a number of clear current candidates:

-   regex (http://pypi.python.org/pypi/regex) - approved by Guido[1].
-   daemon (PEP 3143)
-   ipaddr (PEP 3144)

Other possible future use cases include:

-   Improved HTTP modules (e.g. requests)
-   HTML 5 parsing support (e.g. html5lib)
-   Improved URL/URI/IRI parsing
-   A standard image API (PEP 368)
-   Improved encapsulation of import state (PEP 406)
-   Standard event loop API (PEP 3153)
-   A binary version of WSGI for Python 3 (e.g. PEP 444)
-   Generic function support (e.g. simplegeneric)

Rejected alternatives and variations

See PEP 408.

References

Copyright

This document has been placed in the public domain.

[1] https://mail.python.org/pipermail/python-dev/2012-January/115962.html