PEP: 655 Title: Marking individual TypedDict items as required or
potentially-missing Author: David Foster <david at dafoster.net>
Sponsor: Guido van Rossum <guido at python.org> Discussions-To:
https://mail.python.org/archives/list/typing-sig@python.org/thread/53XVOD5ZUKJ263MWA6AUPEA6J7LBBLNV/
Status: Final Type: Standards Track Topic: Typing Created: 30-Jan-2021
Python-Version: 3.11 Post-History: 31-Jan-2021, 11-Feb-2021,
20-Feb-2021, 26-Feb-2021, 17-Jan-2022, 28-Jan-2022 Resolution:
https://mail.python.org/archives/list/python-dev@python.org/message/AJEDNVC3FXM5QXNNW5CR4UCT4KI5XVUE/

typing:required-notrequired, typing.Required and typing.NotRequired

Abstract

PEP 589 defines notation for declaring a TypedDict with all required
keys and notation for defining a TypedDict with
all potentially-missing keys <589#totality>, however it does not provide
a mechanism to declare some keys as required and others as
potentially-missing. This PEP introduces two new notations: Required[],
which can be used on individual items of a TypedDict to mark them as
required, and NotRequired[], which can be used on individual items to
mark them as potentially-missing.

This PEP makes no Python grammar changes. Correct usage of required and
potentially-missing keys of TypedDicts is intended to be enforced only
by static type checkers and need not be enforced by Python itself at
runtime.

Motivation

It is not uncommon to want to define a TypedDict with some keys that are
required and others that are potentially-missing. Currently the only way
to define such a TypedDict is to declare one TypedDict with one value
for total and then inherit it from another TypedDict with a different
value for total:

    class _MovieBase(TypedDict):  # implicitly total=True
        title: str

    class Movie(_MovieBase, total=False):
        year: int

Having to declare two different TypedDict types for this purpose is
cumbersome.

This PEP introduces two new type qualifiers, typing.Required and
typing.NotRequired, which allow defining a single TypedDict with a mix
of both required and potentially-missing keys:

    class Movie(TypedDict):
        title: str
        year: NotRequired[int]

This PEP also makes it possible to define TypedDicts in the
alternative functional syntax <589#alternative-syntax> with a mix of
required and potentially-missing keys, which is not currently possible
at all because the alternative syntax does not support inheritance:

    Actor = TypedDict('Actor', {
        'name': str,
        # "in" is a keyword, so the functional syntax is necessary
        'in': NotRequired[List[str]],
    })

Rationale

One might think it unusual to propose notation that prioritizes marking
required keys rather than potentially-missing keys, as is customary in
other languages like TypeScript:

    interface Movie {
        title: string;
        year?: number;  // ? marks potentially-missing keys
    }

The difficulty is that the best word for marking a potentially-missing
key, Optional[], is already used in Python for a completely different
purpose: marking values that could be either of a particular type or
None. In particular the following does not work:

    class Movie(TypedDict):
        ...
        year: Optional[int]  # means int|None, not potentially-missing!

Attempting to use any synonym of “optional” to mark potentially-missing
keys (like Missing[]) would be too similar to Optional[] and be easy to
confuse with it.

Thus it was decided to focus on positive-form phrasing for required keys
instead, which is straightforward to spell as Required[].

Nevertheless it is common for folks wanting to extend a regular
(total=True) TypedDict to only want to add a small number of
potentially-missing keys, which necessitates a way to mark keys that are
not required and potentially-missing, and so we also allow the
NotRequired[] form for that case.

Specification

The typing.Required type qualifier is used to indicate that a variable
declared in a TypedDict definition is a required key:

    class Movie(TypedDict, total=False):
        title: Required[str]
        year: int

Additionally the typing.NotRequired type qualifier is used to indicate
that a variable declared in a TypedDict definition is a
potentially-missing key:

    class Movie(TypedDict):  # implicitly total=True
        title: str
        year: NotRequired[int]

It is an error to use Required[] or NotRequired[] in any location that
is not an item of a TypedDict. Type checkers must enforce this
restriction.

It is valid to use Required[] and NotRequired[] even for items where it
is redundant, to enable additional explicitness if desired:

    class Movie(TypedDict):
        title: Required[str]  # redundant
        year: NotRequired[int]

It is an error to use both Required[] and NotRequired[] at the same
time:

    class Movie(TypedDict):
        title: str
        year: NotRequired[Required[int]]  # ERROR

Type checkers must enforce this restriction. The runtime implementations
of Required[] and NotRequired[] may also enforce this restriction.

The alternative functional syntax <589#alternative-syntax> for TypedDict
also supports Required[] and NotRequired[]:

    Movie = TypedDict('Movie', {'name': str, 'year': NotRequired[int]})

Interaction with total=False

Any PEP 589-style TypedDict declared with total=False is equivalent to a
TypedDict with an implicit total=True definition with all of its keys
marked as NotRequired[].

Therefore:

    class _MovieBase(TypedDict):  # implicitly total=True
        title: str

    class Movie(_MovieBase, total=False):
        year: int

is equivalent to:

    class _MovieBase(TypedDict):
        title: str

    class Movie(_MovieBase):
        year: NotRequired[int]

Interaction with Annotated[]

Required[] and NotRequired[] can be used with Annotated[], in any
nesting order:

    class Movie(TypedDict):
        title: str
        year: NotRequired[Annotated[int, ValueRange(-9999, 9999)]]  # ok

    class Movie(TypedDict):
        title: str
        year: Annotated[NotRequired[int], ValueRange(-9999, 9999)]  # ok

In particular allowing Annotated[] to be the outermost annotation for an
item allows better interoperability with non-typing uses of annotations,
which may always want Annotated[] as the outermost annotation. [1]

Runtime behavior

Interaction with get_type_hints()

typing.get_type_hints(...) applied to a TypedDict will by default strip
out any Required[] or NotRequired[] type qualifiers, since these
qualifiers are expected to be inconvenient for code casually
introspecting type annotations.

typing.get_type_hints(..., include_extras=True) however will retain
Required[] and NotRequired[] type qualifiers, for advanced code
introspecting type annotations that wishes to preserve all annotations
in the original source:

    class Movie(TypedDict):
        title: str
        year: NotRequired[int]

    assert get_type_hints(Movie) == \
        {'title': str, 'year': int}
    assert get_type_hints(Movie, include_extras=True) == \
        {'title': str, 'year': NotRequired[int]}

Interaction with get_origin() and get_args()

typing.get_origin() and typing.get_args() will be updated to recognize
Required[] and NotRequired[]:

    assert get_origin(Required[int]) is Required
    assert get_args(Required[int]) == (int,)

    assert get_origin(NotRequired[int]) is NotRequired
    assert get_args(NotRequired[int]) == (int,)

Interaction with __required_keys__ and __optional_keys__

An item marked with Required[] will always appear in the
__required_keys__ for its enclosing TypedDict. Similarly an item marked
with NotRequired[] will always appear in __optional_keys__.

    assert Movie.__required_keys__ == frozenset({'title'})
    assert Movie.__optional_keys__ == frozenset({'year'})

Backwards Compatibility

No backward incompatible changes are made by this PEP.

How to Teach This

To define a TypedDict where most keys are required and some are
potentially-missing, define a single TypedDict as normal (without the
total keyword) and mark those few keys that are potentially-missing with
NotRequired[].

To define a TypedDict where most keys are potentially-missing and a few
are required, define a total=False TypedDict and mark those few keys
that are required with Required[].

If some items accept None in addition to a regular value, it is
recommended that the TYPE|None notation be preferred over Optional[TYPE]
for marking such item values, to avoid using Required[] or NotRequired[]
alongside Optional[] within the same TypedDict definition:

Yes:

    from __future__ import annotations  # for Python 3.7-3.9

    class Dog(TypedDict):
        name: str
        owner: NotRequired[str|None]

Okay (required for Python 3.5.3-3.6):

    class Dog(TypedDict):
        name: str
        owner: 'NotRequired[str|None]'

No:

    class Dog(TypedDict):
        name: str
        # ick; avoid using both Optional and NotRequired
        owner: NotRequired[Optional[str]]

Usage in Python <3.11

If your code supports Python <3.11 and wishes to use Required[] or
NotRequired[] then it should use typing_extensions.TypedDict rather than
typing.TypedDict because the latter will not understand (Not)Required[].
In particular __required_keys__ and __optional_keys__ on the resulting
TypedDict type will not be correct:

Yes (Python 3.11+ only):

    from typing import NotRequired, TypedDict

    class Dog(TypedDict):
        name: str
        owner: NotRequired[str|None]

Yes (Python <3.11 and 3.11+):

    from __future__ import annotations  # for Python 3.7-3.9

    from typing_extensions import NotRequired, TypedDict  # for Python <3.11 with (Not)Required

    class Dog(TypedDict):
        name: str
        owner: NotRequired[str|None]

No (Python <3.11 and 3.11+):

    from typing import TypedDict  # oops: should import from typing_extensions instead
    from typing_extensions import NotRequired

    class Movie(TypedDict):
        title: str
        year: NotRequired[int]

    assert Movie.__required_keys__ == frozenset({'title', 'year'})  # yikes
    assert Movie.__optional_keys__ == frozenset()  # yikes

Reference Implementation

The mypy 0.930, pyright 1.1.117, and pyanalyze 0.4.0 type checkers
support Required and NotRequired.

A reference implementation of the runtime component is provided in the
typing_extensions module.

Rejected Ideas

Special syntax around the key of a TypedDict item

    class MyThing(TypedDict):
        opt1?: str  # may not exist, but if exists, value is string
        opt2: Optional[str]  # always exists, but may have None value

This notation would require Python grammar changes and it is not
believed that marking TypedDict items as required or potentially-missing
would meet the high bar required to make such grammar changes.

    class MyThing(TypedDict):
        Optional[opt1]: str  # may not exist, but if exists, value is string
        opt2: Optional[str]  # always exists, but may have None value

This notation causes Optional[] to take on different meanings depending
on where it is positioned, which is inconsistent and confusing.

Also, “let’s just not put funny syntax before the colon.”[2]

Marking required or potentially-missing keys with an operator

We could use unary + as shorthand to mark a required key, unary - to
mark a potentially-missing key, or unary ~ to mark a key with
opposite-of-normal totality:

    class MyThing(TypedDict, total=False):
        req1: +int    # + means a required key, or Required[]
        opt1: str
        req2: +float

    class MyThing(TypedDict):
        req1: int
        opt1: -str    # - means a potentially-missing key, or NotRequired[]
        req2: float

    class MyThing(TypedDict):
        req1: int
        opt1: ~str    # ~ means a opposite-of-normal-totality key
        req2: float

Such operators could be implemented on type via the __pos__, __neg__ and
__invert__ special methods without modifying the grammar.

It was decided that it would be prudent to introduce long-form notation
(i.e. Required[] and NotRequired[]) before introducing any short-form
notation. Future PEPs may reconsider introducing this or other
short-form notation options.

Note when reconsidering introducing this short-form notation that +, -,
and ~ already have existing meanings in the Python typing world:
covariant, contravariant, and invariant:

    >>> from typing import TypeVar
    >>> (TypeVar('T', covariant=True), TypeVar('U', contravariant=True), TypeVar('V'))
    (+T, -U, ~V)

Marking absence of a value with a special constant

We could introduce a new type-level constant which signals the absence
of a value when used as a union member, similar to JavaScript’s
undefined type, perhaps called Missing:

    class MyThing(TypedDict):
        req1: int
        opt1: str|Missing
        req2: float

Such a Missing constant could also be used for other scenarios such as
the type of a variable which is only conditionally defined:

    class MyClass:
        attr: int|Missing

        def __init__(self, set_attr: bool) -> None:
            if set_attr:
                self.attr = 10

    def foo(set_attr: bool) -> None:
        if set_attr:
            attr = 10
        reveal_type(attr)  # int|Missing

Misalignment with how unions apply to values

However this use of ...|Missing, equivalent to Union[..., Missing],
doesn’t align well with what a union normally means: Union[...] always
describes the type of a value that is present. By contrast missingness
or non-totality is a property of a variable instead. Current precedent
for marking properties of a variable include Final[...] and
ClassVar[...], which the proposal for Required[...] is aligned with.

Misalignment with how unions are subdivided

Furthermore the use of Union[..., Missing] doesn’t align with the usual
ways that union values are broken down: Normally you can eliminate
components of a union type using isinstance checks:

    class Packet:
        data: Union[str, bytes]

    def send_data(packet: Packet) -> None:
        if isinstance(packet.data, str):
            reveal_type(packet.data)  # str
            packet_bytes = packet.data.encode('utf-8')
        else:
            reveal_type(packet.data)  # bytes
            packet_bytes = packet.data
        socket.send(packet_bytes)

However if we were to allow Union[..., Missing] you’d either have to
eliminate the Missing case with hasattr for object attributes:

    class Packet:
        data: Union[str, Missing]

    def send_data(packet: Packet) -> None:
        if hasattr(packet, 'data'):
            reveal_type(packet.data)  # str
            packet_bytes = packet.data.encode('utf-8')
        else:
            reveal_type(packet.data)  # Missing? error?
            packet_bytes = b''
        socket.send(packet_bytes)

or a check against locals() for local variables:

    def send_data(packet_data: Optional[str]) -> None:
        packet_bytes: Union[str, Missing]
        if packet_data is not None:
            packet_bytes = packet.data.encode('utf-8')

        if 'packet_bytes' in locals():
            reveal_type(packet_bytes)  # bytes
            socket.send(packet_bytes)
        else:
            reveal_type(packet_bytes)  # Missing? error?

or a check via other means, such as against globals() for global
variables:

    warning: Union[str, Missing]
    import sys
    if sys.version_info < (3, 6):
        warning = 'Your version of Python is unsupported!'

    if 'warning' in globals():
        reveal_type(warning)  # str
        print(warning)
    else:
        reveal_type(warning)  # Missing? error?

Weird and inconsistent. Missing is not really a value at all; it’s an
absence of definition and such an absence should be treated specially.

Difficult to implement

Eric Traut from the Pyright type checker team has stated that
implementing a Union[..., Missing]-style notation would be difficult.[3]

Introduces a second null-like value into Python

Defining a new Missing type-level constant would be very close to
introducing a new Missing value-level constant at runtime, creating a
second null-like runtime value in addition to None. Having two different
null-like constants in Python (None and Missing) would be confusing.
Many newcomers to JavaScript already have difficulty distinguishing
between its analogous constants null and undefined.

Replace Optional with Nullable. Repurpose Optional to mean “optional item”.

Optional[] is too ubiquitous to deprecate, although use of it may fade
over time in favor of the T|None notation specified by PEP 604.

Change Optional to mean “optional item” in certain contexts instead of “nullable”

Consider the use of a special flag on a TypedDict definition to alter
the interpretation of Optional inside the TypedDict to mean “optional
item” rather than its usual meaning of “nullable”:

    class MyThing(TypedDict, optional_as_missing=True):
        req1: int
        opt1: Optional[str]

or:

    class MyThing(TypedDict, optional_as_nullable=False):
        req1: int
        opt1: Optional[str]

This would add more confusion for users because it would mean that in
some contexts the meaning of Optional[] is different than in other
contexts, and it would be easy to overlook the flag.

Various synonyms for “potentially-missing item”

-   Omittable – too easy to confuse with optional
-   OptionalItem, OptionalKey – two words; too easy to confuse with
    optional
-   MayExist, MissingOk – two words
-   Droppable – too similar to Rust’s Drop, which means something
    different
-   Potential – too vague
-   Open – sounds like applies to an entire structure rather then to an
    item
-   Excludable
-   Checked

References

Copyright

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.

[1] https://bugs.python.org/issue46491

[2] https://mail.python.org/archives/list/typing-sig@python.org/message/4I3GPIWDUKV6GUCHDMORGUGRE4F4SXGR/

[3] https://mail.python.org/archives/list/typing-sig@python.org/message/S2VJSVG6WCIWPBZ54BOJPG56KXVSLZK6/