PEP: 466 Title: Network Security Enhancements for Python 2.7.x Version:
$Revision$ Last-Modified: $Date$ Author: Alyssa Coghlan
<ncoghlan@gmail.com>, Status: Final Type: Standards Track Content-Type:
text/x-rst Created: 23-Mar-2014 Python-Version: 2.7.9 Post-History:
23-Mar-2014, 24-Mar-2014, 25-Mar-2014, 26-Mar-2014, 16-Apr-2014
Resolution:
https://mail.python.org/pipermail/python-dev/2014-April/134163.html

Abstract

Most CPython tracker issues are classified as errors in behaviour or
proposed enhancements. Most patches to fix behavioural errors are
applied to all active maintenance branches. Enhancement patches are
restricted to the default branch that becomes the next Python version.

This cadence works reasonably well during Python's normal 18-24 month
feature release cycle, which is still applicable to the Python 3 series.
However, the age of the standard library in Python 2 has now reached a
point where it is sufficiently far behind the state of the art in
network security protocols for it to be causing real problems in use
cases where upgrading to Python 3 in the near term may not be feasible.

In recognition of the additional practical considerations that have
arisen during the 4+ year maintenance cycle for Python 2.7, this PEP
allows a critical set of network security related features to be
backported from Python 3.4 to upcoming Python 2.7.x maintenance
releases.

While this PEP does not make any changes to the core development team's
handling of security-fix-only branches that are no longer in active
maintenance, it does recommend that commercial redistributors providing
extended support periods for the Python standard library either backport
these features to their supported versions, or else explicitly disclaim
support for the use of older versions in roles that involve connecting
directly to the public internet.

New security related features in Python 2.7 maintenance releases

Under this proposal, the following features will be backported from
Python 3.4 to upcoming Python 2.7.x maintenance releases:

-   in the os module:
    -   persistent file descriptor for os.urandom().
-   in the hmac module:
    -   constant time comparison function (hmac.compare_digest()).
-   in the hashlib module:
    -   password hashing function (hashlib.pbkdf2_hmac()).
    -   details of hash algorithm availability
        (hashlib.algorithms_guaranteed and
        hashlib.algorithms_available).
-   in the ssl module:
    -   this module is almost entirely synchronised with its Python 3
        counterpart, bringing TLSv1.x settings, SSLContext manipulation,
        Server Name Indication, access to platform certificate stores,
        standard library support for peer hostname validation and more
        to the Python 2 series.
    -   the only ssl module features not backported under this policy
        are the ssl.RAND_* functions that provide access to OpenSSL's
        random number generation capabilities - use os.urandom()
        instead.

As a general change in maintenance policy, permission is also granted to
upgrade to newer feature releases of OpenSSL when preparing the binary
installers for new maintenance releases of Python 2.7.

This PEP does NOT propose a general exception for backporting new
features to Python 2.7 - every new feature proposed for backporting will
still need to be justified independently. In particular, it will need to
be explained why relying on an independently updated backport on the
Python Package Index instead is not an acceptable solution.

Implementation status

This PEP originally proposed adding all listed features to the Python
2.7.7 maintenance release. That approach proved to be too ambitious
given the limited time frame between the original creation and
acceptance of the PEP and the release of Python 2.7.7rc1. Instead, the
progress of each individual accepted feature backport is being tracked
as an independent enhancement targeting Python 2.7.

Implemented for Python 2.7.7:

-   Issue #21306: backport hmac.compare_digest
-   Issue #21462: upgrade OpenSSL in the Python 2.7 Windows installers

Implemented for Python 2.7.8:

-   Issue #21304: backport hashlib.pbkdf2

Implemented for Python 2.7.9 (in development):

-   Issue #21308: backport specified ssl module features
-   Issue #21307: backport remaining specified hashlib module features
-   Issue #21305: backport os.urandom shared file descriptor change

Backwards compatibility considerations

As in the Python 3 series, the backported ssl.create_default_context()
API is granted a backwards compatibility exemption that permits the
protocol, options, cipher and other settings of the created SSL context
to be updated in maintenance releases to use higher default security
settings. This allows them to appropriately balance compatibility and
security at the time of the maintenance release, rather than at the time
of the original feature release.

This PEP does not grant any other exemptions to the usual backwards
compatibility policy for maintenance releases. Instead, by explicitly
encouraging the use of feature based checks, it is designed to make it
easier to write more secure cross-version compatible Python software,
while still limiting the risk of breaking currently working software
when upgrading to a new Python 2.7 maintenance release.

In all cases where this proposal allows new features to be backported to
the Python 2.7 release series, it is possible to write cross-version
compatible code that operates by "feature detection" (for example,
checking for particular attributes in a module), without needing to
explicitly check the Python version.

It is then up to library and framework code to provide an appropriate
warning and fallback behaviour if a desired feature is found to be
missing. While some especially security sensitive software MAY fail
outright if a desired security feature is unavailable, most software
SHOULD instead emit a warning and continue operating using a slightly
degraded security configuration.

The backported APIs allow library and application code to perform the
following actions after detecting the presence of a relevant network
security related feature:

-   explicitly opt in to more secure settings (to allow the use of
    enhanced security features in older maintenance releases of Python
    with less secure default behaviour)
-   explicitly opt in to less secure settings (to allow the use of newer
    Python feature releases in lower security environments)
-   determine the default setting for the feature (this MAY require
    explicit Python version checks to determine the Python feature
    release, but DOES NOT require checking for a specific maintenance
    release)

Security related changes to other modules (such as higher level
networking libraries and data format processing libraries) will continue
to be made available as backports and new modules on the Python Package
Index, as independent distribution remains the preferred approach to
handling software that must continue to evolve to handle changing
development requirements independently of the Python 2 standard library.
Refer to the Motivation and Rationale section for a review of the
characteristics that make the secure networking infrastructure worthy of
special consideration.

OpenSSL compatibility

Under this proposal, OpenSSL may be upgraded to more recent feature
releases in Python 2.7 maintenance releases. On Linux and most other
POSIX systems, the specific version of OpenSSL used already varies, as
CPython dynamically links to the system provided OpenSSL library by
default.

For the Windows binary installers, the _ssl and _hashlib modules are
statically linked with OpenSSL and the associated symbols are not
exported. Marc-Andre Lemburg indicates that updating to newer OpenSSL
releases in the egenix-pyopenssl binaries has not resulted in any
reported compatibility issues[1]

The Mac OS X binary installers historically followed the same policy as
other POSIX installations and dynamically linked to the Apple provided
OpenSSL libraries. However, Apple has now ceased updating these
cross-platform libraries, instead requiring that even cross-platform
developers adopt Mac OS X specific interfaces to access up to date
security infrastructure on their platform. Accordingly, and
independently of this PEP, the Mac OS X binary installers were already
going to be switched to statically linker newer versions of OpenSSL[2]

Other Considerations

Maintainability

A number of developers, including Alex Gaynor and Donald Stufft, have
expressed interest in carrying out the feature backports covered by this
policy, and assisting with any additional maintenance burdens that arise
in the Python 2 series as a result.

Steve Dower and Brian Curtin have offered to help with the creation of
the Windows installers, allowing Martin von Löwis the opportunity to
step back from the task of maintaining the 2.7 Windows installer.

This PEP is primarily about establishing the consensus needed to allow
them to carry out this work. For other core developers, this policy
change shouldn't impose any additional effort beyond potentially
reviewing the resulting patches for those developers specifically
interested in the affected modules.

Security releases

This PEP does not propose any changes to the handling of security
releases - those will continue to be source only releases that include
only critical security fixes.

However, the recommendations for library and application developers are
deliberately designed to accommodate commercial redistributors that
choose to apply these changes to additional Python release series that
are either in security fix only mode, or have been declared "end of
life" by the core development team.

Whether or not redistributors choose to exercise that option will be up
to the individual redistributor.

Integration testing

Third party integration testing services should offer users the ability
to test against multiple Python 2.7 maintenance releases (at least 2.7.6
and 2.7.7+), to ensure that libraries, frameworks and applications can
still test their handling of the legacy security infrastructure
correctly (either failing or degrading gracefully, depending on the
security sensitivity of the software), even after the features covered
in this proposal have been backported to the Python 2.7 series.

Handling lower security environments with low risk tolerance

For better or for worse (mostly worse), there are some environments
where the risk of latent security defects is more tolerated than even a
slightly increased risk of regressions in maintenance releases. This
proposal largely excludes these environments from consideration where
the modules covered by the exemption are concerned - this approach is
entirely inappropriate for software connected to the public internet,
and defence in depth security principles suggest that it is not
appropriate for most private networks either.

Downstream redistributors may still choose to cater to such
environments, but they will need to handle the process of downgrading
the security related modules and doing the associated regression testing
themselves. The main CPython continuous integration infrastructure will
not cover this scenario.

Motivation and Rationale

The creation of this PEP was prompted primarily by the aging SSL support
in the Python 2 series. As of March 2014, the Python 2.7 SSL module is
approaching four years of age, and the SSL support in the still popular
Python 2.6 release had its feature set locked six years ago.

These are simply too old to provide a foundation that can be recommended
in good conscience for secure networking software that operates over the
public internet, especially in an era where it is becoming quite clearly
evident that advanced persistent security threats are even more
widespread and more indiscriminate in their targeting than had
previously been understood. While they represented reasonable security
infrastructure in their time, the state of the art has moved on, and we
need to investigate mechanisms for effectively providing more up to date
network security infrastructure for users that, for whatever reason, are
not currently in a position to migrate to Python 3.

While the use of the system OpenSSL installation addresses many of these
concerns on Linux platforms, it doesn't address all of them (in
particular, it is still difficult for sotware to explicitly require some
higher level security settings). The standard library support can be
bypassed by using a third party library like PyOpenSSL or Pycurl, but
this still results in a security problem, as these can be difficult
dependencies to deploy, and many users will remain unaware that they
might want them. Rather than explaining to potentially naive users how
to obtain and use these libraries, it seems better to just fix the
included batteries.

In the case of the binary installers for Windows and Mac OS X that are
published on python.org, the version of OpenSSL used is entirely within
the control of the Python core development team, but is currently
limited to OpenSSL maintenance releases for the version initially
shipped with the corresponding Python feature release.

With increased popularity comes increased responsibility, and this
proposal aims to acknowledge the fact that Python's popularity and
adoption is at a sufficiently high level that some of our design and
policy decisions have significant implications beyond the Python
development community.

As one example, the Python 2 ssl module does not support the Server Name
Indication standard. While it is possible to obtain SNI support by using
the third party requests client library, actually doing so currently
requires using not only requests and its embedded dependencies, but also
half a dozen or more additional libraries. The lack of support in the
Python 2 series thus serves as an impediment to making effective use of
SNI on servers, as Python 2 clients will frequently fail to handle it
correctly.

Another more critical example is the lack of SSL hostname matching in
the Python 2 standard library - it is currently necessary to rely on a
third party library, such as requests or backports.ssl_match_hostname to
obtain that functionality in Python 2.

The Python 2 series also remains more vulnerable to remote timing
attacks on security sensitive comparisons than the Python 3 series, as
it lacks a standard library equivalent to the timing attack resistant
hmac.compare_digest() function. While appropriate secure comparison
functions can be implemented in third party extensions, many users don't
even consider the issue and use ordinary equality comparisons instead -
while a standard library solution doesn't automatically fix that
problem, it does make the barrier to resolution much lower once the
problem is pointed out.

Python 2.7 represents the only long term maintenance release the core
development team has provided, and it is natural that there will be
things that worked over a historically shorter maintenance lifespan that
don't work over this longer support period. In the specific case of the
problem described in this PEP, the simplest available solution is to
acknowledge that long term maintenance of network security related
modules requires the ability to add new features, even while retaining
backwards compatibility for existing interfaces.

For those familiar with it, it is worth comparing the approach described
in this PEP with Red Hat's handling of its long term open source support
commitments: it isn't the RHEL 6.0 release itself that receives 10 years
worth of support, but the overall RHEL 6 series. The individual RHEL 6.x
point releases within the series then receive a wide variety of new
features, including security enhancements, all while meeting strict
backwards compatibility guarantees for existing software. The proposal
covered in this PEP brings our approach to long term maintenance more
into line with this precedent - we retain our strict backwards
compatibility requirements, but make an exception to the restriction
against adding new features.

To date, downstream redistributors have respected our upstream policy of
"no new features in Python maintenance releases". This PEP explicitly
accepts that a more nuanced policy is appropriate in the case of network
security related features, and the specific change it describes is
deliberately designed such that it is potentially suitable for Red Hat
Enterprise Linux and its downstream derivatives.

Why these particular changes?

The key requirement for a feature to be considered for inclusion in this
proposal was that it must have security implications beyond the specific
application that is written in Python and the system that application is
running on. Thus the focus on network security protocols, password
storage and related cryptographic infrastructure - Python is a popular
choice for the development of web services and clients, and thus the
capabilities of widely used Python versions have implications for the
security design of other services that may themselves be using newer
versions of Python or other development languages, but need to
interoperate with clients or servers written using older versions of
Python.

The intent behind this requirement was to minimise any impact that the
introduction of this policy may have on the stability and compatibility
of maintenance releases, while still addressing some key security
concerns relating to the particular aspects of Python 2.7. It would be
thoroughly counterproductive if end users became as cautious about
updating to new Python 2.7 maintenance releases as they are about
updating to new feature releases within the same release series.

The ssl module changes are included in this proposal to bring the Python
2 series up to date with the past 4 years of evolution in network
security standards, and make it easier for those standards to be broadly
adopted in both servers and clients. Similarly the hash algorithm
availability indicators in hashlib are included to make it easier for
applications to detect and employ appropriate hash definitions across
both Python 2 and 3.

The hmac.compare_digest() and hashlib.pbkdf2_hmac() are included to help
lower the barriers to secure password storage and checking in Python 2
server applications.

The os.urandom() change has been included in this proposal to further
encourage users to leave the task of providing high quality random
numbers for cryptographic use cases to operating system vendors. The use
of insufficiently random numbers has the potential to compromise any
cryptographic system, and operating system developers have more tools
available to address that problem adequately than the typical Python
application runtime.

Rejected alternative: just advise developers to migrate to Python 3

This alternative represents the status quo. Unfortunately, it has proven
to be unworkable in practice, as the backwards compatibility
implications mean that this is a non-trivial migration process for large
applications and integration projects. While the tools for migration
have evolved to a point where it is possible to migrate even large
applications opportunistically and incrementally (rather than all at
once) by updating code to run in the large common subset of Python 2 and
Python 3, using the most recent technology often isn't a priority in
commercial environments.

Previously, this was considered an acceptable harm, as while it was an
unfortunate problem for the affected developers to have to face, it was
seen as an issue between them and their management chain to make the
case for infrastructure modernisation, and this case would become
naturally more compelling as the Python 3 series evolved.

However, now that we're fully aware of the impact the limitations of the
Python 2 standard library may be having on the evolution of internet
security standards, I no longer believe that it is reasonable to expect
platform and application developers to resolve all of the latent defects
in an application's Unicode correctness solely in order to gain access
to the network security enhancements already available in Python 3.

While Ubuntu (and to some extent Debian as well) are committed to
porting all default system services and scripts to Python 3, and to
removing Python 2 from its default distribution images (but not from its
archives), this is a mammoth task and won't be completed for the Ubuntu
14.04 LTS release (at least for the desktop image - it may be achieved
for the mobile and server images).

Fedora has even more work to do to migrate, and it will take a
non-trivial amount of time to migrate the relevant infrastructure
components. While Red Hat are also actively working to make it easier
for users to use more recent versions of Python on our stable platforms,
it's going to take time for those efforts to start having an impact on
end users' choice of version, and any such changes also don't benefit
the core platform infrastructure that runs in the integrated system
Python by necessity.

The OpenStack migration to Python 3 is also still in its infancy, and
even though that's a project with an extensive and relatively robust
automated test suite, it's still large enough that it is going to take
quite some time to migrate fully to a Python 2/3 compatible code base.

And that's just three of the highest profile open source projects that
make heavy use of Python. Given the likely existence of large amounts of
legacy code that lacks the kind of automated regression test suite
needed to help support a migration from Python 2 to Python 3, there are
likely to be many cases where reimplementation (perhaps even in Python
3) proves easier than migration. The key point of this PEP is that those
situations affect more people than just the developers and users of the
affected application: the existence of clients and servers with outdated
network security infrastructure becomes something that developers of
secure networked services need to take into account as part of their
security design, and that's a problem that inhibits the adoption of
better security standards.

As Terry Reedy noted, if we try to persist with the status quo, the
likely outcome is that commercial redistributors will attempt to do
something like this on behalf of their customers anyway, but in a
potentially inconsistent and ad hoc manner. By drawing the scope
definition process into the upstream project we are in a better position
to influence the approach taken to address the situation and to help
ensure some consistency across redistributors.

The problem is real, so something needs to change, and this PEP
describes my preferred approach to addressing the situation.

Rejected alternative: create and release Python 2.8

With sufficient corporate support, it likely would be possible to create
and release Python 2.8 (it's highly unlikely such a project would garner
enough interest to be achievable with only volunteers). However, this
wouldn't actually solve the problem, as the aim is to provide a
relatively low impact way to incorporate enhanced security features into
integrated products and deployments that make use of Python 2.

Upgrading to a new Python feature release would mean both more work for
the core development team, as well as a more disruptive update that most
potential end users would likely just skip entirely.

Attempting to create a Python 2.8 release would also bring in
suggestions to backport many additional features from Python 3 (such as
tracemalloc and the improved coroutine support), making the migration
from Python 2.7 to this hypothetical 2.8 release even riskier and more
disruptive.

This is not a recommended approach, as it would involve substantial
additional work for a result that is actually less effective in
achieving the original aim (which is to eliminate the current widespread
use of the aging network security infrastructure in the Python 2
series).

Furthermore, while I can't make any commitments to actually addressing
this issue on Red Hat platforms, I can categorically rule out the idea
of a Python 2.8 being of any use to me in even attempting to get it
addressed.

Rejected alternative: distribute the security enhancements via PyPI

While this initially appears to be an attractive and easier to manage
approach, it actually suffers from several significant problems.

Firstly, this is complex, low level, cross-platform code that integrates
with the underlying operating system across a variety of POSIX platforms
(including Mac OS X) and Windows. The CPython BuildBot fleet is already
set up to handle continuous integration in that context, but most of the
freely available continuous integration services just offer Linux, and
perhaps paid access to Windows. Those services work reasonably well for
software that largely runs on the abstraction layers offered by Python
and other dynamic languages, as well as the more comprehensive
abstraction offered by the JVM, but won't suffice for the kind of code
involved here.

The OpenSSL dependency for the network security support also qualifies
as the kind of "complex binary dependency" that isn't yet handled well
by the pip based software distribution ecosystem. Relying on a third
party binary dependency also creates potential compatibility problems
for pip when running on other interpreters like PyPy.

Another practical problem with the idea is the fact that pip itself
relies on the ssl support in the standard library (with some additional
support from a bundled copy of requests, which in turn bundles
backport.ssl_match_hostname), and hence would require any replacement
module to also be bundled within pip. This wouldn't pose any
insurmountable difficulties (it's just another dependency to vendor),
but it would mean yet another copy of OpenSSL to keep up to date.

This approach also has the same flaw as all other "improve security by
renaming things" approaches: they completely miss the users who most
need help, and raise significant barriers against being able to
encourage users to do the right thing when their infrastructure supports
it (since "use this other module" is a much higher impact change than
"turn on this higher security setting"). Deprecating the aging SSL
infrastructure in the standard library in favour of an external module
would be even more user hostile than accepting the slightly increased
risk of regressions associated with upgrading it in place.

Last, but certainly not least, this approach suffers from the same
problem as the idea of doing a Python 2.8 release: likely not solving
the actual problem. Commercial redistributors of Python are set up to
redistribute Python, and a pre-existing set of additional packages.
Getting new packages added to the pre-existing set can be done, but
means approaching each and every redistributor and asking them to update
their repackaging process accordingly. By contrast, the approach
described in this PEP would require redistributors to deliberately opt
out of the security enhancements by deliberately downgrading the
provided network security infrastructure, which most of them are
unlikely to do.

Rejected variant: provide a "legacy SSL infrastructure" branch

Earlier versions of this PEP included the concept of a 2.7-legacy-ssl
branch that preserved the exact feature set of the Python 2.7.6 network
security infrastructure.

In my opinion, anyone that actually wants this is almost certainly
making a mistake, and if they insist they really do want it in their
specific situation, they're welcome to either make it themselves or
arrange for a downstream redistributor to make it for them.

If they are made publicly available, any such rebuilds should be
referred to as "Python 2.7 with Legacy SSL" to clearly distinguish them
from the official Python 2.7 releases that include more up to date
network security infrastructure.

After the first Python 2.7 maintenance release that implements this PEP,
it would also be appropriate to refer to Python 2.7.6 and earlier
releases as "Python 2.7 with Legacy SSL".

Rejected variant: synchronise particular modules entirely with Python 3

Earlier versions of this PEP suggested synchronising the hmac, hashlib
and ssl modules entirely with their Python 3 counterparts.

This approach proved too vague to build a compelling case for the
exception, and has thus been replaced by the current more explicit
proposal.

Rejected variant: open ended backport policy

Earlier versions of this PEP suggested a general policy change related
to future Python 3 enhancements that impact the general security of the
internet.

That approach created unnecessary uncertainty, so it has been simplified
to propose backport a specific concrete set of changes. Future feature
backport proposals can refer back to this PEP as precedent, but it will
still be necessary to make a specific case for each feature addition to
the Python 2.7 long-term support release.

Disclosure of Interest

The author of this PEP currently works for Red Hat on test automation
tools. If this proposal is accepted, I will be strongly encouraging Red
Hat to take advantage of the resulting opportunity to help improve the
overall security of the Python ecosystem. However, I do not speak for
Red Hat in this matter, and cannot make any commitments on Red Hat's
behalf.

Acknowledgements

Thanks to Christian Heimes and other for their efforts in greatly
improving Python's SSL support in the Python 3 series, and a variety of
members of the Python community for helping me to better understand the
implications of the default settings we provide in our SSL modules, and
the impact that tolerating the use of SSL infrastructure that was
defined in 2010 (Python 2.7) or even 2008 (Python 2.6) potentially has
for the security of the web as a whole.

Thanks to Donald Stufft and Alex Gaynor for identifying a more limited
set of essential security features that allowed the proposal to be made
more fine-grained than backporting entire modules from Python 3.4
([3],[4]).

Christian and Donald also provided valuable feedback on a preliminary
draft of this proposal.

Thanks also to participants in the python-dev mailing list threads
([5],[6],[7],[8]), as well as the various folks I discussed this issue
with at PyCon 2014 in Montreal.

References

Copyright

This document has been placed in the public domain.



  Local Variables: mode: indented-text indent-tabs-mode: nil
  sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

[1] Marc-Andre Lemburg's OpenSSL feedback for Windows
(https://mail.python.org/pipermail/python-dev/2014-March/133438.html)

[2] Ned Deily's OpenSSL feedback for Mac OS X
(https://mail.python.org/pipermail/python-dev/2014-March/133347.html)

[3] Donald Stufft's recommended set of backported features
(https://mail.python.org/pipermail/python-dev/2014-March/133500.html)

[4] Alex Gaynor's recommended set of backported features
(https://mail.python.org/pipermail/python-dev/2014-March/133503.html)

[5] PEP 466 discussion (round 1)
(https://mail.python.org/pipermail/python-dev/2014-March/133334.html)

[6] PEP 466 discussion (round 2)
(https://mail.python.org/pipermail/python-dev/2014-March/133389.html)

[7] PEP 466 discussion (round 3)
(https://mail.python.org/pipermail/python-dev/2014-March/133442.html)

[8] PEP 466 discussion (round 4)
(https://mail.python.org/pipermail/python-dev/2014-March/133472.html)