PEP: 698 Title: Override Decorator for Static Typing Author: Steven
Troxler <steven.troxler@gmail.com>, Joshua Xu <jxu425@fb.com>, Shannon
Zhu <szhu@fb.com> Sponsor: Jelle Zijlstra <jelle.zijlstra at gmail.com>
Discussions-To:
https://discuss.python.org/t/pep-698-a-typing-override-decorator/20839
Status: Final Type: Standards Track Topic: Typing Created: 05-Sep-2022
Python-Version: 3.12 Post-History: 20-May-2022, 17-Aug-2022,
11-Oct-2022, 07-Nov-2022, Resolution:
https://discuss.python.org/t/pep-698-a-typing-override-decorator/20839/11

typing:override and @typing.override <typing.override>

Abstract

This PEP proposes adding an @override decorator to the Python type
system. This will allow type checkers to prevent a class of bugs that
occur when a base class changes methods that are inherited by derived
classes.

Motivation

A primary purpose of type checkers is to flag when refactors or changes
break pre-existing semantic structures in the code, so users can
identify and make fixes across their project without doing a manual
audit of their code.

Safe Refactoring

Python’s type system does not provide a way to identify call sites that
need to be changed to stay consistent when an overridden function API
changes. This makes refactoring and transforming code more dangerous.

Consider this simple inheritance structure:

    class Parent:
        def foo(self, x: int) -> int:
            return x

    class Child(Parent):
        def foo(self, x: int) -> int:
            return x + 1

    def parent_callsite(parent: Parent) -> None:
        parent.foo(1)

    def child_callsite(child: Child) -> None:
        child.foo(1)

If the overridden method on the superclass is renamed or deleted, type
checkers will only alert us to update call sites that deal with the base
type directly. But the type checker can only see the new code, not the
change we made, so it has no way of knowing that we probably also needed
to rename the same method on child classes.

A type checker will happily accept this code, even though we are likely
introducing bugs:

    class Parent:
        # Rename this method
        def new_foo(self, x: int) -> int:
            return x

    class Child(Parent):
        # This (unchanged) method used to override `foo` but is unrelated to `new_foo`
        def foo(self, x: int) -> int:
            return x + 1

    def parent_callsite(parent: Parent) -> None:
        # If we pass a Child instance we’ll now run Parent.new_foo - likely a bug
        parent.new_foo(1)

    def child_callsite(child: Child) -> None:
        # We probably wanted to invoke new_foo here. Instead, we forked the method
        child.foo(1)

This code will type check, but there are two potential sources of bugs:

-   If we pass a Child instance to the parent_callsite function, it will
    invoke the implementation in Parent.new_foo. rather than Child.foo.
    This is probably a bug - we presumably would not have written
    Child.foo in the first place if we didn’t need custom behavior.
-   Our system was likely relying on Child.foo behaving in a similar way
    to Parent.foo. But unless we catch this early, we have now forked
    the methods, and in future refactors it is likely no one will
    realize that major changes to the behavior of new_foo likely require
    updating Child.foo as well, which could lead to major bugs later.

The incorrectly-refactored code is type-safe, but is probably not what
we intended and could cause our system to behave incorrectly. The bug
can be difficult to track down because our new code likely does execute
without throwing exceptions. Tests are less likely to catch the problem,
and silent errors can take longer to track down in production.

We are aware of several production outages in multiple typed codebases
caused by such incorrect refactors. This is our primary motivation for
adding an @override decorator to the type system, which lets developers
express the relationship between Parent.foo and Child.foo so that type
checkers can detect the problem.

Rationale

Subclass Implementations Become More Explicit

We believe that explicit overrides will make unfamiliar code easier to
read than implicit overrides. A developer reading the implementation of
a subclass that uses @override can immediately see which methods are
overriding functionality in some base class; without this decorator, the
only way to quickly find out is using a static analysis tool.

Precedent in Other Languages and Runtime Libraries

Static Override Checks in Other Languages

Many popular programming languages support override checks. For example:

-   C++ has override.
-   C# has override.
-   Hack has <<__Override>>.
-   Java has @Override.
-   Kotlin has override.
-   Scala has override.
-   Swift has override.
-   TypeScript has override.

Runtime Override Checks in Python

Today, there is an Overrides library that provides decorators @overrides
[sic] and @final and will enforce them at runtime.

PEP 591 added a @final decorator with the same semantics as those in the
Overrides library. But the override component of the runtime library is
not supported statically at all, which has added some confusion around
the mix/matched support.

Providing support for @override in static checks would add value
because:

-   Bugs can be caught earlier, often in-editor.
-   Static checks come with no performance overhead, unlike runtime
    checks.
-   Bugs will be caught quickly even in rarely-used modules, whereas
    with runtime checks these might go undetected for a time without
    automated tests of all imports.

Disadvantages

Using @override will make code more verbose.

Specification

When type checkers encounter a method decorated with @typing.override
they should treat it as a type error unless that method is overriding a
compatible method or attribute in some ancestor class.

    from typing import override

    class Parent:
        def foo(self) -> int:
            return 1

        def bar(self, x: str) -> str:
            return x

    class Child(Parent):
        @override
        def foo(self) -> int:
            return 2

        @override
        def baz(self) -> int:  # Type check error: no matching signature in ancestor
            return 1

The @override decorator should be permitted anywhere a type checker
considers a method to be a valid override, which typically includes not
only normal methods but also @property, @staticmethod, and @classmethod.

No New Rules for Override Compatibility

This PEP is exclusively concerned with the handling of the new @override
decorator, which specifies that the decorated method must override some
attribute in an ancestor class. This PEP does not propose any new rules
regarding the type signatures of such methods.

Strict Enforcement Per-Project

We believe that @override is most useful if checkers also allow
developers to opt into a strict mode where methods that override a
parent class are required to use the decorator. Strict enforcement
should be opt-in for backward compatibility.

Motivation

The primary reason for a strict mode that requires @override is that
developers can only trust that refactors are override-safe if they know
that the @override decorator is used throughout the project.

There is another class of bug related to overrides that we can only
catch using a strict mode.

Consider the following code:

    class Parent:
        pass

    class Child(Parent):
        def foo(self) -> int:
            return 2

Imagine we refactor it as follows:

    class Parent:
        def foo(self) -> int:   # This method is new
            return 1

    class Child(Parent):
        def foo(self) -> int:  # This is now an override!
            return 2

    def call_foo(parent: Parent) -> int:
        return parent.foo()  # This could invoke Child.foo, which may be surprising.

The semantics of our code changed here, which could cause two problems:

-   If the author of the code change did not know that Child.foo already
    existed (which is very possible in a large codebase), they might be
    surprised to see that call_foo does not always invoke Parent.foo.
-   If the codebase authors tried to manually apply @override everywhere
    when writing overrides in subclasses, they are likely to miss the
    fact that Child.foo needs it here.

At first glance this kind of change may seem unlikely, but it can
actually happen often if one or more subclasses have functionality that
developers later realize belongs in the base class.

With a strict mode, we will always alert developers when this occurs.

Precedent

Most of the typed, object-oriented programming languages we looked at
have an easy way to require explicit overrides throughout a project:

-   C#, Kotlin, Scala, and Swift always require explicit overrides
-   TypeScript has a --no-implicit-override flag to force explicit
    overrides
-   In Hack and Java the type checker always treats overrides as opt-in,
    but widely-used linters can warn if explicit overrides are missing.

Backward Compatibility

By default, the @override decorator will be opt-in. Codebases that do
not use it will type-check as before, without the additional type
safety.

Runtime Behavior

Set __override__ = True when possible

At runtime, @typing.override will make a best-effort attempt to add an
attribute __override__ with value True to its argument. By "best-effort"
we mean that we will try adding the attribute, but if that fails (for
example because the input is a descriptor type with fixed slots) we will
silently return the argument as-is.

This is exactly what the @typing.final decorator does, and the
motivation is similar: it gives runtime libraries the ability to use
@override. As a concrete example, a runtime library could check
__override__ in order to automatically populate the __doc__ attribute of
child class methods using the parent method docstring.

Limitations of setting __override__

As described above, adding __override__ may fail at runtime, in which
case we will simply return the argument as-is.

In addition, even in cases where it does work, it may be difficult for
users to correctly work with multiple decorators, because successfully
ensuring the __override__ attribute is set on the final output requires
understanding the implementation of each decorator:

-   The @override decorator needs to execute after ordinary decorators
    like @functools.lru_cache that use wrapper functions, since we want
    to set __override__ on the outermost wrapper. This means it needs to
    go above all these other decorators.
-   But @override needs to execute before many special descriptor-based
    decorators like @property, @staticmethod, and @classmethod.
-   As discussed above, in some cases (for example a descriptor with
    fixed slots or a descriptor that also wraps) it may be impossible to
    set the __override__ attribute at all.

As a result, runtime support for setting __override__ is best effort
only, and we do not expect type checkers to validate the ordering of
decorators.

Rejected Alternatives

Rely on Integrated Development Environments for safety

Modern Integrated Development Environments (IDEs) often provide the
ability to automatically update subclasses when renaming a method. But
we view this as insufficient for several reasons:

-   If a codebase is split into multiple projects, an IDE will not help
    and the bug appears when upgrading dependencies. Type checkers are a
    fast way to catch breaking changes in dependencies.
-   Not all developers use such IDEs. And library maintainers, even if
    they do use an IDE, should not need to assume pull request authors
    use the same IDE. We prefer being able to detect problems in
    continuous integration without assuming anything about developers’
    choice of editor.

Runtime enforcement

We considered having @typing.override enforce override safety at
runtime, similarly to how @overrides.overrides does today.

We rejected this for four reasons:

-   For users of static type checking, it is not clear this brings any
    benefits.
-   There would be at least some performance overhead, leading to
    projects importing slower with runtime enforcement. We estimate the
    @overrides.overrides implementation takes around 100 microseconds,
    which is fast but could still add up to a second or more of extra
    initialization time in million-plus line codebases, which is exactly
    where we think @typing.override will be most useful.
-   An implementation may have edge cases where it doesn’t work well (we
    heard from a maintainer of one such closed-source library that this
    has been a problem). We expect static enforcement to be simple and
    reliable.
-   The implementation approaches we know of are not simple. The
    decorator executes before the class is finished evaluating, so the
    options we know of are either to inspect the bytecode of the caller
    (as @overrides.overrides does) or to use a metaclass-based approach.
    Neither approach seems ideal.

Mark a base class to force explicit overrides on subclasses

We considered including a class decorator @require_explicit_overrides,
which would have provided a way for base classes to declare that all
subclasses must use the @override decorator on method overrides. The
Overrides library has a mixin class, EnforceExplicitOverrides, which
provides similar behavior in runtime checks.

We decided against this because we expect owners of large codebases will
benefit most from @override, and for these use cases having a strict
mode where explicit @override is required (see the Backward
Compatibility section) provides more benefits than a way to mark base
classes.

Moreover we believe that authors of projects who do not consider the
extra type safety to be worth the additional boilerplate of using
@override should not be forced to do so. Having an optional strict mode
puts the decision in the hands of project owners, whereas the use of
@require_explicit_overrides in libraries would force project owners to
use @override even if they prefer not to.

Include the name of the ancestor class being overridden

We considered allowing the caller of @override to specify a specific
ancestor class where the overridden method should be defined:

    class Parent0:
        def foo(self) -> int:
            return 1


    class Parent1:
        def bar(self) -> int:
            return 1


    class Child(Parent0, Parent1):
        @override(Parent0)  # okay, Parent0 defines foo
        def foo(self) -> int:
            return 2

        @override(Parent0)  # type error, Parent0 does not define bar
        def bar(self) -> int:
            return 2

This could be useful for code readability because it makes the override
structure more explicit for deep inheritance trees. It also might catch
bugs by prompting developers to check that the implementation of an
override still makes sense whenever a method being overridden moves from
one base class to another.

We decided against it because:

-   Supporting this would add complexity to the implementation of both
    @override and type checker support for it, so there would need to be
    considerable benefits.
-   We believe that it would be rarely used and catch relatively few
    bugs.
    -   The author of the Overrides package has noted that early
        versions of his library included this capability but it was
        rarely useful and seemed to have little benefit. After it was
        removed, the ability was never requested by users.

Reference Implementation

Pyre: A proof of concept is implemented in Pyre:

-   The decorator @pyre_extensions.override can mark overrides
-   Pyre can type-check this decorator as specified in this PEP

Copyright

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.