PEP: 493 Title: HTTPS verification migration tools for Python 2.7
Version: $Revision$ Last-Modified: $Date$ Author: Alyssa Coghlan
<ncoghlan@gmail.com>, Robert Kuska <rkuska@redhat.com>, Marc-André
Lemburg <mal@lemburg.com> BDFL-Delegate: Barry Warsaw Status: Final
Type: Standards Track Content-Type: text/x-rst Created: 10-May-2015
Python-Version: 2.7.12 Post-History: 06-Jul-2015, 11-Nov-2015,
24-Nov-2015, 24-Feb-2016 Resolution:
https://mail.python.org/pipermail/python-dev/2016-March/143450.html

Abstract

PEP 476 updated Python's default handling of HTTPS certificates in
client modules to align with certificate handling in web browsers, by
validating that the certificates received belonged to the server the
client was attempting to contact. The Python 2.7 long term maintenance
series was judged to be in scope for this change, with the new behaviour
introduced in the Python 2.7.9 maintenance release.

This has created a non-trivial barrier to adoption for affected Python
2.7 maintenance releases, so this PEP proposes additional Python 2.7
specific features that allow system administrators and other users to
more easily decouple the decision to verify server certificates in HTTPS
client modules from the decision to update to newer Python 2.7
maintenance releases.

Rationale

PEP 476 changed Python's default behaviour to align with expectations
established by web browsers in regards to the semantics of HTTPS URLs:
starting with Python 2.7.9 and 3.4.3, HTTPS clients in the standard
library validate server certificates by default.

However, it is also the case that this change does cause problems for
infrastructure administrators operating private intranets that rely on
self-signed certificates, or otherwise encounter problems with the new
default certificate verification settings.

To manage these kinds of situations, web browsers provide users with
"click through" warnings that allow the user to add the server's
certificate to the browser's certificate store. Network client tools
like curl and wget offer options to switch off certificate checking
entirely (by way of curl --insecure and wget --no-check-certificate,
respectively).

At a different layer of the technology stack, Linux security modules
like SELinux and AppArmor, while enabled by default by distribution
vendors, offer relatively straightforward mechanisms for turning them
off.

At the moment, no such convenient mechanisms exist to disable Python's
default certificate checking for a whole process.

PEP 476 did attempt to address this question, by covering how to revert
to the old settings process wide by monkeypatching the ssl module to
restore the old behaviour. Unfortunately, the sitecustomize.py based
technique proposed to allow system administrators to disable the feature
by default in their Standard Operating Environment definition has been
determined to be insufficient in at least some cases. The specific case
that led to the initial creation of this PEP is the one where a Linux
distributor aims to provide their users with a smoother migration path
than the standard one provided by consuming upstream CPython 2.7
releases directly, but other potential challenges have also been pointed
out with updating embedded Python runtimes and other user level
installations of Python.

Rather than allowing a plethora of mutually incompatible migration
techniques to bloom, this PEP proposes an additional feature to be added
to Python 2.7.12 to make it easier to revert a process to the past
behaviour of skipping certificate validation in HTTPS client modules. It
also provides additional recommendations to redistributors backporting
these features to versions of Python prior to Python 2.7.9.

Alternatives

In the absence of clear upstream guidance and recommendations,
commercial redistributors will still make their own design decisions in
the interests of their customers. The main approaches available are:

-   Continuing to rebase on new Python 2.7.x releases, while providing
    no additional assistance beyond the mechanisms defined in PEP 476 in
    migrating from unchecked to checked hostnames in standard library
    HTTPS clients
-   Gating availability of the changes in default handling of HTTPS
    connections on upgrading from Python 2 to Python 3
-   For Linux distribution vendors, gating availability of the changes
    in default handling of HTTPS connections on upgrading to a new
    operating system version
-   Implementing one or both of the backport suggestions described in
    this PEP, regardless of the formal status of the PEP

Scope Limitations

These changes are being proposed purely as tools for helping to manage
the transition to the new default certificate handling behaviour in the
context of Python 2.7. They are not being proposed as new features for
Python 3, as it is expected that the vast majority of client
applications affected by this problem without the ability to update the
application itself will be Python 2 applications.

It would likely be desirable for a future version of Python 3 to allow
the default certificate handling for secure protocols to be configurable
on a per-protocol basis, but that question is beyond the scope of this
PEP.

Requirements for capability detection

As the proposals in this PEP aim to facilitate backports to earlier
Python versions, the Python version number cannot be used as a reliable
means for detecting them. Instead, they are designed to allow the
presence or absence of the feature to be determined using the following
technique:

    python -c "import ssl; ssl.<_relevant_attribute>"

This will fail with AttributeError (and hence a non-zero return code) if
the relevant capability is not available.

The feature detection attributes defined by this PEP are:

-   ssl._https_verify_certificates: runtime configuration API
-   ssl._https_verify_envvar: environment based configuration
-   ssl._cert_verification_config: file based configuration (PEP 476
    opt-in)

The marker attributes are prefixed with an underscore to indicate the
implementation dependent and security sensitive nature of these
capabilities.

Feature: Configuration API

This change is proposed for inclusion in CPython 2.7.12 and later
CPython 2.7.x releases. It consists of a new
ssl._https_verify_certificates() to specify the default handling of
HTTPS certificates in standard library client libraries.

It is not proposed to forward port this change to Python 3, so Python 3
applications that need to support skipping certificate verification will
still need to define their own suitable security context.

Feature detection

The marker attribute on the ssl module related to this feature is the
ssl._https_verify_certificates function itself.

Specification

The ssl._https_verify_certificates function will work as follows:

    def _https_verify_certificates(enable=True):
        """Verify server HTTPS certificates by default?"""
        global _create_default_https_context
        if enable:
            _create_default_https_context = create_default_context
        else:
            _create_default_https_context = _create_unverified_context

If called without arguments, or with enable set to a true value, then
standard library client modules will subsequently verify HTTPS
certificates by default, otherwise they will skip verification.

If called with enable set to a false value, then standard library client
modules will subsequently skip verifying HTTPS certificates by default.

Security Considerations

The inclusion of this feature will allow security sensitive applications
to include the following forward-compatible snippet in their code:

    if hasattr(ssl, "_https_verify_certificates"):
        ssl._https_verify_certificates()

Some developers may also choose to opt out of certificate checking using
ssl._https_verify_certificates(enable=False). This doesn't introduce any
major new security concerns, as monkeypatching the affected internal
APIs was already possible.

Feature: environment based configuration

This change is proposed for inclusion in CPython 2.7.12 and later
CPython 2.7.x releases. It consists of a new PYTHONHTTPSVERIFY
environment variable that can be set to '0' to disable the default
verification without modifying the application source code (which may
not even be available in cases of bytecode-only application
distribution)

It is not proposed to forward port this change to Python 3, so Python 3
applications that need to support skipping certificate verification will
still need to define their own suitable security context.

Feature detection

The marker attribute on the ssl module related to this feature is:

-   the ssl._https_verify_envvar attribute, giving the name of
    environment variable affecting the default behaviour

This not only makes it straightforward to detect the presence (or
absence) of the capability, it also makes it possible to
programmatically determine the relevant environment variable name.

Specification

Rather than always defaulting to the use of ssl.create_default_context,
the ssl module will be modified to:

-   read the PYTHONHTTPSVERIFY environment variable when the module is
    first imported into a Python process
-   set the ssl._create_default_https_context function to be an alias
    for ssl._create_unverified_context if this environment variable is
    present and set to '0'
-   otherwise, set the ssl._create_default_https_context function to be
    an alias for ssl.create_default_context as usual

Example implementation

    _https_verify_envvar = 'PYTHONHTTPSVERIFY'

    def _get_https_context_factory():
        if not sys.flags.ignore_environment:
            config_setting = os.environ.get(_https_verify_envvar)
            if config_setting == '0':
                return _create_unverified_context
        return create_default_context

    _create_default_https_context = _get_https_context_factory()

Security Considerations

Relative to the behaviour in Python 3.4.3+ and Python 2.7.9->2.7.11,
this approach does introduce a new downgrade attack against the default
security settings that potentially allows a sufficiently determined
attacker to revert Python to the default behaviour used in CPython 2.7.8
and earlier releases.

This slight increase in the available attack surface is a key reason
why:

-   security sensitive applications should still define their own SSL
    context
-   the migration features described in this PEP are not being added to
    Python 3

However, it's also worth keeping in mind that carrying out such an
attack requires the ability to modify the execution environment of a
Python process prior to the import of the ssl module. In combination
with the ability to write to any part of the filesystem (such as /tmp),
any attacker with such access would already be able to modify the
behaviour of the underlying OpenSSL implementation, the dynamic library
loader, and other potentially security sensitive components.

Interaction with Python virtual environments

The default setting is read directly from the process environment, and
hence works the same way regardless of whether or not the interpreter is
being run inside an activated Python virtual environment.

Reference Implementation

A patch for Python 2.7 implementing the above two features is attached
to the relevant tracker issue.

Backporting this PEP to earlier Python versions

If this PEP is accepted, then commercial Python redistributors may
choose to backport the per-process configuration mechanisms defined in
this PEP to base versions older than Python 2.7.9, without also
backporting PEP 476's change to the default behaviour of the overall
Python installation.

Such a backport would differ from the mechanism proposed in this PEP
solely in the default behaviour when PYTHONHTTPSVERIFY was not set at
all: it would continue to default to skipping certificate validation.

In this case, if the PYTHONHTTPSVERIFY environment variable is defined,
and set to anything other than '0', then HTTPS certificate verification
should be enabled.

Feature detection

There's no specific attribute indicating that this situation applies.
Rather, it is indicated by the ssl._https_verify_certificates and
ssl._https_verify_envvar attributes being present in a Python version
that is nominally older than Python 2.7.12.

Specification

Implementing this backport involves backporting the changes in PEP 466,
476 and this PEP, with the following change to the handling of the
PYTHONHTTPSVERIFY environment variable in the ssl module:

-   read the PYTHONHTTPSVERIFY environment variable when the module is
    first imported into a Python process
-   set the ssl._create_default_https_context function to be an alias
    for ssl.create_default_context if this environment variable is
    present and set to any value other than '0'
-   otherwise, set the ssl._create_default_https_context function to be
    an alias for ssl._create_unverified_context

Example implementation

    _https_verify_envvar = 'PYTHONHTTPSVERIFY'

    def _get_https_context_factory():
        if not sys.flags.ignore_environment:
            config_setting = os.environ.get(_https_verify_envvar)
            if config_setting != '0':
                return create_default_context
        return _create_unverified_context

    _create_default_https_context = _get_https_context_factory()

    def _disable_https_default_verification():
        """Skip verification of HTTPS certificates by default"""
        global _create_default_https_context
        _create_default_https_context = _create_unverified_context

Security Considerations

This change would be a strict security upgrade for any Python version
that currently defaults to skipping certificate validation in standard
library HTTPS clients. The technical trade-offs to be taken into account
relate largely to the magnitude of the PEP 466 backport also required
rather than to anything security related.

Interaction with Python virtual environments

The default setting is read directly from the process environment, and
hence works the same way regardless of whether or not the interpreter is
being run inside an activated Python virtual environment.

Backporting PEP 476 to earlier Python versions

The backporting approach described above leaves the default HTTPS
certificate verification behaviour of a Python 2.7 installation
unmodified: verifying certificates still needs to be opted into on a
per-connection or per-process basis.

To allow the default behaviour of the entire installation to be modified
without breaking backwards compatibility, Red Hat designed a
configuration mechanism for the system Python 2.7 installation in Red
Hat Enterprise Linux 7.2+ that provides:

-   an opt-in model that allows the decision to enable HTTPS certificate
    verification to be made independently of the decision to upgrade to
    the operating system version where the feature was first backported
-   the ability for system administrators to set the default behaviour
    of Python applications and scripts run directly in the system Python
    installation
-   the ability for the redistributor to consider changing the default
    behaviour of new installations at some point in the future without
    impacting existing installations that have been explicitly
    configured to skip verifying HTTPS certificates by default

As it only affects backports to earlier releases of Python 2.7, this
change is not proposed for inclusion in upstream CPython, but rather is
offered as a recommendation to other redistributors that choose to offer
a similar feature to their users.

This PEP doesn't take a position on whether or not this particular
change is a good idea - rather, it suggests that if a redistributor
chooses to go down the path of making the default behaviour configurable
in a version of Python older than Python 2.7.9, then maintaining a
consistent approach across redistributors would be beneficial for users.

However, this approach SHOULD NOT be used for any Python installation
that advertises itself as providing Python 2.7.9 or later, as most
Python users will have the reasonable expectation that all such
environments will verify HTTPS certificates by default.

Feature detection

The marker attribute on the ssl module related to this feature is:

    _cert_verification_config = '<path to configuration file>'

This not only makes it straightforward to detect the presence (or
absence) of the capability, it also makes it possible to
programmatically determine the relevant configuration file name.

Recommended modifications to the Python standard library

The recommended approach to backporting the PEP 476 modifications to an
earlier point release is to implement the following changes relative to
the default PEP 476 behaviour implemented in Python 2.7.9+:

-   modify the ssl module to read a system wide configuration file when
    the module is first imported into a Python process
-   define a platform default behaviour (either verifying or not
    verifying HTTPS certificates) to be used if this configuration file
    is not present
-   support selection between the following three modes of operation:
    -   ensure HTTPS certificate verification is enabled
    -   ensure HTTPS certificate verification is disabled
    -   delegate the decision to the redistributor providing this Python
        version
-   set the ssl._create_default_https_context function to be an alias
    for either ssl.create_default_context or
    ssl._create_unverified_context based on the given configuration
    setting.

Recommended file location

As the PEP authors are not aware of any vendors providing long-term
support releases targeting Windows, Mac OS X or *BSD systems, this
approach is currently only specifically defined for Linux system Python
installations.

The recommended configuration file name on Linux systems is
/etc/python/cert-verification.cfg.

The .cfg filename extension is recommended for consistency with the
pyvenv.cfg used by the venv module in Python 3's standard library.

Recommended file format

The configuration file should use a ConfigParser ini-style format with a
single section named [https] containing one required setting verify.

The suggested section name is taken from the "https" URL schema passed
to affected client APIs.

Permitted values for verify are:

-   enable: ensure HTTPS certificate verification is enabled by default
-   disable: ensure HTTPS certificate verification is disabled by
    default
-   platform_default: delegate the decision to the redistributor
    providing this particular Python version

If the [https] section or the verify setting are missing, or if the
verify setting is set to an unknown value, it should be treated as if
the configuration file is not present.

Example implementation

    _cert_verification_config = '/etc/python/cert-verification.cfg'

    def _get_https_context_factory():
        # Check for a system-wide override of the default behaviour
        context_factories = {
            'enable': create_default_context,
            'disable': _create_unverified_context,
            'platform_default': _create_unverified_context, # For now :)
        }
        import ConfigParser
        config = ConfigParser.RawConfigParser()
        config.read(_cert_verification_config)
        try:
            verify_mode = config.get('https', 'verify')
        except (ConfigParser.NoSectionError, ConfigParser.NoOptionError):
            verify_mode = 'platform_default'
        default_factory = context_factories.get('platform_default')
        return context_factories.get(verify_mode, default_factory)

    _create_default_https_context = _get_https_context_factory()

Security Considerations

The specific recommendations for this backporting case are designed to
work for privileged, security sensitive processes, even those being run
in the following locked down configuration:

-   run from a locked down administrator controlled directory rather
    than a normal user directory (preventing sys.path[0] based privilege
    escalation attacks)
-   run using the -E switch (preventing PYTHON* environment variable
    based privilege escalation attacks)
-   run using the -s switch (preventing user site directory based
    privilege escalation attacks)
-   run using the -S switch (preventing sitecustomize based privilege
    escalation attacks)

The intent is that the only reason HTTPS verification should be getting
turned off installation wide when using this approach is because:

-   an end user is running a redistributor provided version of CPython
    rather than running upstream CPython directly
-   that redistributor has decided to provide a smoother migration path
    to verifying HTTPS certificates by default than that being provided
    by the upstream project
-   either the redistributor or the local infrastructure administrator
    has determined that it is appropriate to retain the default
    pre-2.7.9 behaviour (at least for the time being)

Using an administrator controlled configuration file rather than an
environment variable has the essential feature of providing a smoother
migration path, even for applications being run with the -E switch.

Interaction with Python virtual environments

This setting is scoped by the interpreter installation and affects all
Python processes using that interpreter, regardless of whether or not
the interpreter is being run inside an activated Python virtual
environment.

Origins of this recommendation

This recommendation is based on the backporting approach adopted for Red
Hat Enterprise Linux 7.2, as published in the original July 2015 draft
of this PEP and described in detail in this KnowledgeBase article. Red
Hat's patches implementing this backport for Python 2.7.5 can be found
in the CentOS git repository.

Recommendation for combined feature backports

If a redistributor chooses to backport the environment variable based
configuration setting from this PEP to a modified Python version that
also implements the configuration file based PEP 476 backport, then the
environment variable should take precedence over the system-wide
configuration setting. This allows the setting to be changed for a given
user or application, regardless of the installation-wide default
behaviour.

Example implementation

    _https_verify_envvar = 'PYTHONHTTPSVERIFY'
    _cert_verification_config = '/etc/python/cert-verification.cfg'

    def _get_https_context_factory():
        # Check for an environmental override of the default behaviour
        if not sys.flags.ignore_environment:
            config_setting = os.environ.get(_https_verify_envvar)
            if config_setting is not None:
                if config_setting == '0':
                    return _create_unverified_context
                return create_default_context

        # Check for a system-wide override of the default behaviour
        context_factories = {
            'enable': create_default_context,
            'disable': _create_unverified_context,
            'platform_default': _create_unverified_context, # For now :)
        }
        import ConfigParser
        config = ConfigParser.RawConfigParser()
        config.read(_cert_verification_config)
        try:
            verify_mode = config.get('https', 'verify')
        except (ConfigParser.NoSectionError, ConfigParser.NoOptionError):
            verify_mode = 'platform_default'
        default_factory = context_factories.get('platform_default')
        return context_factories.get(verify_mode, default_factory)

    _create_default_https_context = _get_https_context_factory()

Copyright

This document has been placed into the public domain.