PEP: 565 Title: Show DeprecationWarning in __main__ Author: Alyssa
Coghlan <ncoghlan@gmail.com> Status: Final Type: Standards Track
Content-Type: text/x-rst Created: 12-Nov-2017 Python-Version: 3.7
Post-History: 12-Nov-2017, 25-Nov-2017 Resolution:
https://mail.python.org/pipermail/python-dev/2017-December/151224.html

Abstract

In Python 2.7 and Python 3.2, the default warning filters were updated
to hide DeprecationWarning by default, such that deprecation warnings in
development tools that were themselves written in Python (e.g. linters,
static analysers, test runners, code generators), as well as any other
applications that merely happened to be written in Python, wouldn't be
visible to their users unless those users explicitly opted in to seeing
them.

However, this change has had the unfortunate side effect of making
DeprecationWarning markedly less effective at its primary intended
purpose: providing advance notice of breaking changes in APIs (whether
in CPython, the standard library, or in third party libraries) to users
of those APIs.

To improve this situation, this PEP proposes a single adjustment to the
default warnings filter: displaying deprecation warnings attributed to
the main module by default.

This change will mean that code entered at the interactive prompt and
code in single file scripts will revert to reporting these warnings by
default, while they will continue to be silenced by default for packaged
code distributed as part of an importable module.

The PEP also proposes a number of small adjustments to the reference
interpreter and standard library documentation to help make the warnings
subsystem more approachable for new Python developers.

As part of the documentation updates, it will be made clearer that the
unittest test runner displays all warnings by default when executing
test cases, and that other test runners are advised to follow that
example.

Specification

New default warnings filter entry

The current set of default warnings filters consists of:

    ignore::DeprecationWarning
    ignore::PendingDeprecationWarning
    ignore::ImportWarning
    ignore::BytesWarning
    ignore::ResourceWarning

The default unittest test runner then uses warnings.catch_warnings()
warnings.simplefilter('default') to override the default filters while
running test cases.

The change proposed in this PEP is to update the default warning filter
list to be:

    default::DeprecationWarning:__main__
    ignore::DeprecationWarning
    ignore::PendingDeprecationWarning
    ignore::ImportWarning
    ignore::BytesWarning
    ignore::ResourceWarning

This means that in cases where the nominal location of the warning (as
determined by the stacklevel parameter to warnings.warn) is in the
__main__ module, the first occurrence of each DeprecationWarning will
once again be reported.

This change will lead to DeprecationWarning being displayed by default
for:

-   code executed directly at the interactive prompt
-   code executed directly as part of a single-file script

While continuing to be hidden by default for:

-   code imported from another module in a zipapp archive's __main__.py
    file
-   code imported from another module in an executable package's
    __main__ submodule
-   code imported from an executable script wrapper generated at
    installation time based on a console_scripts or gui_scripts entry
    point definition

This means that tool developers that create an installable or executable
artifact (such as a zipapp archive) for distribution to their users
shouldn't see any change from the status quo, while users of more ad hoc
personal or locally distributed scripts are likely to start seeing
relevant deprecation warnings again (as they did in Python 2.6 and
earlier).

Additional use case for FutureWarning

The standard library documentation will be updated to explicitly
recommend the use of FutureWarning (rather than DeprecationWarning) for
backwards compatibility warnings that are intended to be seen by users
of an application. (This will be in addition to the existing use of
FutureWarning to warn about constructs that will remain valid code in
the future, but will have different semantics).

This will give the following three distinct categories of backwards
compatibility warning, with three different intended audiences:

-   PendingDeprecationWarning: hidden by default for all code. The
    intended audience is Python developers that take an active interest
    in ensuring the future compatibility of their software (e.g.
    professional Python application developers with specific support
    obligations).
-   DeprecationWarning: reported by default for code that runs directly
    in the __main__ module (as such code is considered relatively
    unlikely to have a dedicated test suite), but hidden by default for
    code in other modules. The intended audience is Python developers
    that are at risk of upgrades to their dependencies (including
    upgrades to Python itself) breaking their software (e.g. developers
    using Python to script environments where someone else is in control
    of the timing of dependency upgrades).
-   FutureWarning: reported by default for all code. The intended
    audience is users of applications written in Python, rather than
    other Python developers (e.g. warning about use of a deprecated
    setting in a configuration file format).

For library and framework authors that want to ensure their API
compatibility warnings are more reliably seen by their users, the
recommendation is to use a custom warning class that derives from
DeprecationWarning in Python 3.7+, and from FutureWarning in earlier
versions.

Recommended filter settings for test runners

Developers of test runners are advised to implement logic equivalent to
the following when determining their default warnings filters:

    if not sys.warnoptions:
        warnings.simplefilter("default")

This effectively enables all warnings by default, as if the -Wd command
line option had been passed.

Note that actually enabling BytesWarning in a test suite still requires
passing the -b option to the interpreter at the command line. For
implicit bytes conversion and bytes comparison warnings, the warnings
filter machinery is only used to determine whether they should be
printed as warnings or raised as exceptions - when the command line flag
isn't set, the interpreter doesn't even emit the warning in the first
place.

Recommended filter settings for interactive shells

Developers of interactive shells are advised to add a filter that
enables DeprecationWarning in the namespace where user code is entered
and executed.

If that namespace is __main__ (as it is for the default CPython REPL),
then no changes are needed beyond those in this PEP.

Interactive shell implementations which use a namespace other than
__main__ will need to add their own filter. For example, IPython uses
the following command ([1]) to set up a suitable filter:

    warnings.filterwarnings("default", category=DeprecationWarning,
                                       module=self.user_ns.get("__name__"))

Other documentation updates

The current reference documentation for the warnings system is
relatively short on specific examples of possible settings for the -W
command line option or the PYTHONWARNINGS environment variably that
achieve particular end results.

The following improvements are proposed as part of the implementation of
this PEP:

-   Explicitly list the following entries under the description of the
    PYTHONWARNINGS environment variable:

        PYTHONWARNINGS=error # Convert to exceptions
        PYTHONWARNINGS=always # Warn every time
        PYTHONWARNINGS=default # Warn once per call location
        PYTHONWARNINGS=module # Warn once per calling module
        PYTHONWARNINGS=once # Warn once per Python process
        PYTHONWARNINGS=ignore # Never warn

-   Explicitly list the corresponding short options (-We, -Wa, -Wd, -Wm,
    -Wo, -Wi) for each of the warning actions listed under the -W
    command line switch documentation

-   Explicitly list the default filter set in the warnings module
    documentation, using the action::category and
    action::category:module notation

-   Explicitly list the following snippet in the warnings.simplefilter
    documentation as a recommended approach to turning off all warnings
    by default in a Python application while still allowing them to be
    turned back on via PYTHONWARNINGS or the -W command line switch:

        if not sys.warnoptions:
            warnings.simplefilter("ignore")

None of these are new (they already work in all still supported Python
versions), but they're not especially obvious given the current
structure of the related documentation.

Reference Implementation

A reference implementation is available in the PR[2] linked from the
related tracker issue for this PEP[3].

As a side-effect of implementing this PEP, the internal warnings filter
list will start allowing the use of plain strings as part of filter
definitions (in addition to the existing use of compiled regular
expressions). When present, the plain strings will be compared for exact
matches only. This approach allows the new default filter to be added
during interpreter startup without requiring early access to the re
module.

Motivation

As discussed in[4] and mentioned in[5], Python 2.7 and Python 3.2
changed the default handling of DeprecationWarning such that:

-   the warning was hidden by default during normal code execution
-   the unittest test runner was updated to re-enable it when running
    tests

The intent was to avoid cases of tooling output like the following:

    $ devtool mycode/
    /usr/lib/python3.6/site-packages/devtool/cli.py:1: DeprecationWarning: 'async' and 'await' will become reserved keywords in Python 3.7
      async = True
    ... actual tool output ...

Even when devtool is a tool specifically for Python programmers, this is
not a particularly useful warning, as it will be shown on every
invocation, even though the main helpful step an end user can take is to
report a bug to the developers of devtool.

The warning is even less helpful for general purpose developer tools
that are used across more languages than just Python, and almost
entirely *un*helpful for applications that simply happen to be written
in Python, and aren't necessarily intended for a developer audience at
all.

However, this change proved to have unintended consequences for the
following audiences:

-   anyone using a test runner other than the default one built into
    unittest (the request for third party test runners to change their
    default warnings filters was never made explicitly, so many of them
    still rely on the interpreter defaults that are designed to suit
    deployed applications)
-   anyone using the default unittest test runner to test their Python
    code in a subprocess (since even unittest only adjusts the warnings
    settings in the current process)
-   anyone writing Python code at the interactive prompt or as part of a
    directly executed script that didn't have a Python level test suite
    at all

In these cases, DeprecationWarning ended up become almost entirely
equivalent to PendingDeprecationWarning: it was simply never seen at
all.

Limitations on PEP Scope

This PEP exists specifically to explain both the proposed addition to
the default warnings filter for 3.7, and to more clearly articulate the
rationale for the original change to the handling of DeprecationWarning
back in Python 2.7 and 3.2.

This PEP does not solve all known problems with the current approach to
handling deprecation warnings. Most notably:

-   The default unittest test runner does not currently report
    deprecation warnings emitted at module import time, as the warnings
    filter override is only put in place during test execution, not
    during test discovery and loading.
-   The default unittest test runner does not currently report
    deprecation warnings in subprocesses, as the warnings filter
    override is applied directly to the loaded warnings module, not to
    the PYTHONWARNINGS environment variable.
-   The standard library doesn't provide a straightforward way to opt-in
    to seeing all warnings emitted by a particular dependency prior to
    upgrading it (the third-party warn module[6] does provide this, but
    enabling it involves monkeypatching the standard library's warnings
    module).
-   When software has been factored out into support modules, but those
    modules have little or no automated test coverage, re-enabling
    deprecation warnings by default in __main__ isn't likely to help
    find API compatibility problems. Near term, the best currently
    available answer is to run affected applications with
    PYTHONWARNINGS=default::DeprecationWarning or
    python -W default::DeprecationWarning and pay attention to their
    stderr output. Longer term, this is really a question for
    researchers working on static analysis of Python code: how to
    reliably find usage of deprecated APIs, and how to infer that an API
    or parameter is deprecated based on warnings.warn calls, without
    actually running either the code providing the API or the code
    accessing it.

While these are real problems with the status quo, they're excluded from
consideration in this PEP because they're going to require more complex
solutions than a single additional entry in the default warnings filter,
and resolving them at least potentially won't require going through the
PEP process.

For anyone interested in pursuing them further, the first two would be
unittest module enhancement requests, the third would be a warnings
module enhancement request, while the last would only require a PEP if
inferring API deprecations from their contents was deemed to be an
intractable code analysis problem, and an explicit function and
parameter marker syntax in annotations was proposed instead.

The CPython reference implementation will also include the following
related changes in 3.7:

-   a new -X dev command line option that combines several developer
    centric settings (including -Wd) into one command line flag:
    https://github.com/python/cpython/issues/76224
-   changing the behaviour in debug builds to show more of the warnings
    that are off by default in regular interpreter builds:
    https://github.com/python/cpython/issues/76269

Independently of the proposed changes to the default filters in this
PEP, issue 32229[7] is a proposal to add a warnings.hide_warnings API to
make it simpler for application developers to hide warnings during
normal operation, while easily making them visible when testing.

References

-   First python-dev discussion thread
-   Second python-dev discussion thread

Copyright

This document has been placed in the public domain.

[1] IPython's DeprecationWarning auto-configuration
(https://github.com/ipython/ipython/blob/6.2.x/IPython/core/interactiveshell.py#L619)

[2] GitHub PR for PEP 565 implementation
(https://github.com/python/cpython/pull/4458)

[3] Tracker issue for PEP 565 implementation
(https://github.com/python/cpython/issues/76156)

[4] stdlib-sig thread proposing the original default filter change
(https://mail.python.org/pipermail/stdlib-sig/2009-November/000789.html)

[5] Python 2.7 notification of the default warnings filter change
(https://docs.python.org/3/whatsnew/2.7.html#changes-to-the-handling-of-deprecation-warnings)

[6] Emitting warnings based on the location of the warning itself
(https://pypi.org/project/warn/)

[7] warnings.hide_warnings API proposal
(https://github.com/python/cpython/issues/76410)