PEP: 728 Title: TypedDict with Typed Extra Items Author: Zixuan James Li
<p359101898@gmail.com> Sponsor: Jelle Zijlstra
<jelle.zijlstra@gmail.com> Discussions-To:
https://discuss.python.org/t/pep-728-typeddict-with-typed-extra-items/45443
Status: Draft Type: Standards Track Topic: Typing Content-Type:
text/x-rst Created: 12-Sep-2023 Python-Version: 3.13 Post-History:
09-Feb-2024,

Abstract

This PEP proposes a way to limit extra items for ~typing.TypedDict using
a closed argument and to type them with the special __extra_items__ key.
This addresses the need to define closed TypedDict type or to type a
subset of keys that might appear in a dict while permitting additional
items of a specified type.

Motivation

A typing.TypedDict type can annotate the value type of each known item
in a dictionary. However, due to structural subtyping, a TypedDict can
have extra items that are not visible through its type. There is
currently no way to restrict the types of items that might be present in
the TypedDict type's structural subtypes.

Defining a Closed TypedDict Type

The current behavior of TypedDict prevents users from defining a closed
TypedDict type when it is expected that the type contains no additional
items.

Due to the possible presence of extra items, type checkers cannot infer
more precise return types for .items() and .values() on a TypedDict.
This can also be resolved by defining a closed TypedDict type.

Another possible use case for this is a sound way to enable type
narrowing with the in check:

    class Movie(TypedDict):
        name: str
        director: str

    class Book(TypedDict):
        name: str
        author: str

    def fun(entry: Movie | Book) -> None:
        if "author" in entry:
            reveal_type(entry)  # Revealed type is 'Movie | Book'

Nothing prevents a dict that is structurally compatible with Movie to
have the author key, and under the current specification it would be
incorrect for the type checker to narrow its type.

Allowing Extra Items of a Certain Type

For supporting API interfaces or legacy codebase where only a subset of
possible keys are known, it would be useful to explicitly expect
additional keys of certain value types.

However, the typing spec is more restrictive on type checking the
construction of a TypedDict, preventing users from doing this:

    class MovieBase(TypedDict):
        name: str

    def fun(movie: MovieBase) -> None:
        # movie can have extra items that are not visible through MovieBase
        ...

    movie: MovieBase = {"name": "Blade Runner", "year": 1982}  # Not OK
    fun({"name": "Blade Runner", "year": 1982})  # Not OK

While the restriction is enforced when constructing a TypedDict, due to
structural subtyping, the TypedDict may have extra items that are not
visible through its type. For example:

    class Movie(MovieBase):
        year: int

    movie: Movie = {"name": "Blade Runner", "year": 1982}
    fun(movie)  # OK

It is not possible to acknowledge the existence of the extra items
through in checks and access them without breaking type safety, even
though they might exist from arbitrary structural subtypes of MovieBase:

    def g(movie: MovieBase) -> None:
        if "year" in movie:
            reveal_type(movie["year"])  # Error: TypedDict 'MovieBase' has no key 'year'

Some workarounds have already been implemented in response to the need
to allow extra keys, but none of them is ideal. For mypy,
--disable-error-code=typeddict-unknown-key suppresses type checking
error specifically for unknown keys on TypedDict. This sacrifices type
safety over flexibility, and it does not offer a way to specify that the
TypedDict type expects additional keys compatible with a certain type.

Support Additional Keys for Unpack

PEP 692 adds a way to precisely annotate the types of individual keyword
arguments represented by **kwargs using TypedDict with Unpack. However,
because TypedDict cannot be defined to accept arbitrary extra items, it
is not possible to allow additional keyword arguments that are not known
at the time the TypedDict is defined.

Given the usage of pre-PEP 692 type annotation for **kwargs in existing
codebases, it will be valuable to accept and type extra items on
TypedDict so that the old typing behavior can be supported in
combination with the new Unpack construct.

Rationale

A type that allows extra items of type str on a TypedDict can be loosely
described as the intersection between the TypedDict and
Mapping[str, str].

Index Signatures in TypeScript achieve this:

    type Foo = {
        a: string
        [key: string]: string
    }

This proposal aims to support a similar feature without introducing
general intersection of types or syntax changes, offering a natural
extension to the existing type consistency rules.

We propose that we add an argument closed to TypedDict. Similar to
total, only a literal True or False value is allowed. When closed=True
is used in the TypedDict type definition, we give the dunder attribute
__extra_items__ a special meaning: extra items are allowed, and their
types should be compatible with the value type of __extra_items__.

If closed=True is set, but there is no __extra_items__ key, the
TypedDict is treated as if it contained an item __extra_items__: Never.

Note that __extra_items__ on the same TypedDict type definition will
remain as a regular item if closed=True is not used.

Different from index signatures, the types of the known items do not
need to be consistent with the value type of __extra_items__.

There are some advantages to this approach:

-   Inheritance works naturally. __extra_items__ defined on a TypedDict
    will also be available to its subclasses.
-   We can build on top of the type consistency rules defined in the
    typing spec. __extra_items__ can be treated as a pseudo-item in
    terms of type consistency.
-   There is no need to introduce a grammar change to specify the type
    of the extra items.
-   We can precisely type the extra items without making __extra_items__
    the union of known items.
-   We do not lose backwards compatibility as __extra_items__ still can
    be used as a regular key.

Specification

This specification is structured to parallel PEP 589 to highlight
changes to the original TypedDict specification.

If closed=True is specified, extra items are treated as non-required
items having the same type of __extra_items__ whose keys are allowed
when determining supported and unsupported operations.

Using TypedDict Types

Assuming that closed=True is used in the TypedDict type definition.

For a TypedDict type that has the special __extra_items__ key, during
construction, the value type of each unknown item is expected to be
non-required and compatible with the value type of __extra_items__. For
example:

    class Movie(TypedDict, closed=True):
        name: str
        __extra_items__: bool

    a: Movie = {"name": "Blade Runner", "novel_adaptation": True}  # OK
    b: Movie = {
        "name": "Blade Runner",
        "year": 1982,  # Not OK. 'int' is incompatible with 'bool'
    }  

In this example, __extra_items__: bool does not mean that Movie has a
required string key "__extra_items__" whose value type is bool. Instead,
it specifies that keys other than "name" have a value type of bool and
are non-required.

The alternative inline syntax is also supported:

    Movie = TypedDict("Movie", {"name": str, "__extra_items__": bool}, closed=True)

Accessing extra keys is allowed. Type checkers must infer its value type
from the value type of __extra_items__:

    def f(movie: Movie) -> None:
        reveal_type(movie["name"])              # Revealed type is 'str'
        reveal_type(movie["novel_adaptation"])  # Revealed type is 'bool'

When a TypedDict type defines __extra_items__ without closed=True,
closed defaults to False and the key is assumed to be a regular key:

    class Movie(TypedDict):
        name: str
        __extra_items__: bool

    a: Movie = {"name": "Blade Runner", "novel_adaptation": True}  # Not OK. Unexpected key 'novel_adaptation'
    b: Movie = {
        "name": "Blade Runner",
        "__extra_items__": True,  # OK
    }

For such non-closed TypedDict types, it is assumed that they allow
non-required extra items of value type ReadOnly[object] during
inheritance or type consistency checks. However, extra keys found during
construction should still be rejected by the type checker.

closed is not inherited through subclassing:

    class MovieBase(TypedDict, closed=True):
        name: str
        __extra_items__: ReadOnly[str | None]

    class Movie(MovieBase):
        __extra_items__: str  # A regular key

    a: Movie = {"name": "Blade Runner", "__extra_items__": None}  # Not OK. 'None' is incompatible with 'str'
    b: Movie = {
        "name": "Blade Runner",
        "__extra_items__": "A required regular key",
        "other_extra_key": None,
    }  # OK

Here, "__extra_items__" in a is a regular key defined on Movie where its
value type is narrowed from ReadOnly[str | None] to str,
"other_extra_key" in b is an extra key whose value type must be
consistent with the value type of "__extra_items__" defined on
MovieBase.

Interaction with Totality

It is an error to use Required[] or NotRequired[] with the special
__extra_items__ item. total=False and total=True have no effect on
__extra_items__ itself.

The extra items are non-required, regardless of the totality of the
TypedDict. Operations that are available to NotRequired items should
also be available to the extra items:

    class Movie(TypedDict, closed=True):
        name: str
        __extra_items__: int

    def f(movie: Movie) -> None:
        del movie["name"]  # Not OK
        del movie["year"]  # OK

Interaction with Unpack

For type checking purposes, Unpack[TypedDict] with extra items should be
treated as its equivalent in regular parameters, and the existing rules
for function parameters still apply:

    class Movie(TypedDict, closed=True):
        name: str
        __extra_items__: int

    def f(**kwargs: Unpack[Movie]) -> None: ...

    # Should be equivalent to
    def f(*, name: str, **kwargs: int) -> None: ...

Interaction with PEP 705

When the special __extra_items__ item is annotated with ReadOnly[], the
extra items on the TypedDict have the properties of read-only items.
This interacts with inheritance rules specified in
PEP 705 <705#Inheritance>.

Notably, if the TypedDict type declares __extra_items__ to be read-only,
a subclass of the TypedDict type may redeclare __extra_items__'s value
type or additional non-extra items' value type.

Because a non-closed TypedDict type implicitly allows non-required extra
items of value type ReadOnly[object], its subclass can override the
special __extra_items__ with more specific types.

More details are discussed in the later sections.

Inheritance

When the TypedDict type is defined as closed=False (the default),
__extra_items__ should behave and be inherited the same way a regular
key would. A regular __extra_items__ key can coexist with the special
__extra_items__ and both should be inherited when subclassing.

We assume that closed=True whenever __extra_items__ is mentioned for the
rest of this section.

__extra_items__ is inherited the same way as a regular key: value_type
item. As with the other keys, the same rules from the typing spec and
PEP 705 <705#inheritance> apply. We interpret the existing rules in the
context of __extra_items__.

We need to reinterpret the following rule to define how __extra_items__
interacts with it:

  -   Changing a field type of a parent TypedDict class in a subclass is
      not allowed.

First, it is not allowed to change the value type of __extra_items__ in
a subclass unless it is declared to be ReadOnly in the superclass:

    class Parent(TypedDict, closed=True):
        __extra_items__: int | None

    class Child(Parent, closed=True):
        __extra_items__: int  # Not OK. Like any other TypedDict item, __extra_items__'s type cannot be changed

Second, __extra_items__: T effectively defines the value type of any
unnamed items accepted to the TypedDict and marks them as non-required.
Thus, the above restriction applies to any additional items defined in a
subclass. For each item added in a subclass, all of the following
conditions should apply:

-   If __extra_items__ is read-only
    -   The item can be either required or non-required
    -   The item's value type is consistent with T
-   If __extra_items__ is not read-only
    -   The item is non-required
    -   The item's value type is consistent with T
    -   T is consistent with the item's value type
-   If __extra_items__ is not redeclared, the subclass inherits it
    as-is.

For example:

    class MovieBase(TypedDict, closed=True):
        name: str
        __extra_items__: int | None

    class AdaptedMovie(MovieBase):  # Not OK. 'bool' is not consistent with 'int | None'
        adapted_from_novel: bool

    class MovieRequiredYear(MovieBase):  # Not OK. Required key 'year' is not known to 'Parent'
        year: int | None

    class MovieNotRequiredYear(MovieBase):  # Not OK. 'int | None' is not consistent with 'int'
        year: NotRequired[int]

    class MovieWithYear(MovieBase):  # OK
        year: NotRequired[int | None]

Due to this nature, an important side effect allows us to define a
TypedDict type that disallows additional items:

    class MovieFinal(TypedDict, closed=True):
        name: str
        __extra_items__: Never

Here, annotating __extra_items__ with typing.Never specifies that there
can be no other keys in MovieFinal other than the known ones. Because of
its potential common use, this is equivalent to:

    class MovieFinal(TypedDict, closed=True):
        name: str

where we implicitly assume the __extra_items__: Never field by default
if only closed=True is specified.

Type Consistency

In addition to the set S of keys of the explicitly defined items, a
TypedDict type that has the item __extra_items__: T is considered to
have an infinite set of items that all satisfy the following conditions:

-   If __extra_items__ is read-only
    -   The key's value type is consistent with T
    -   The key is not in S.
-   If __extra_items__ is not read-only
    -   The key is non-required
    -   The key's value type is consistent with T
    -   T is consistent with the key's value type
    -   The key is not in S.

For type checking purposes, let __extra_items__ be a non-required
pseudo-item to be included whenever "for each ... item/key" is stated in
the existing type consistency rules from PEP 705 <705#type-consistency>,
and we modify it as follows:

  A TypedDict type A is consistent with TypedDict B if A is structurally
  compatible with B. This is true if and only if all of the following
  are satisfied:

  -   For each item in B, A has the corresponding key, unless the item
      in B is read-only, not required, and of top value type
      (ReadOnly[NotRequired[object]]). [Edit: Otherwise, if the
      corresponding key with the same name cannot be found in ``A``,
      "__extra_items__" is considered the corresponding key.]
  -   For each item in B, if A has the corresponding key [Edit: or
      "__extra_items__"], the corresponding value type in A is
      consistent with the value type in B.
  -   For each non-read-only item in B, its value type is consistent
      with the corresponding value type in A. [Edit: if the
      corresponding key with the same name cannot be found in ``A``,
      "__extra_items__" is considered the corresponding key.]
  -   For each required key in B, the corresponding key is required in
      A. For each non-required key in B, if the item is not read-only in
      B, the corresponding key is not required in A. [Edit: if the
      corresponding key with the same name cannot be found in ``A``,
      "__extra_items__" is considered to be non-required as the
      corresponding key.]

The following examples illustrate these checks in action.

__extra_items__ puts various restrictions on additional items for type
consistency checks:

    class Movie(TypedDict, closed=True):
        name: str
        __extra_items__: int | None

    class MovieDetails(TypedDict, closed=True):
        name: str
        year: NotRequired[int]
        __extra_items__: int | None

    details: MovieDetails = {"name": "Kill Bill Vol. 1", "year": 2003}
    movie: Movie = details  # Not OK. While 'int' is consistent with 'int | None',
                            # 'int | None' is not consistent with 'int'

    class MovieWithYear(TypedDict, closed=True):
        name: str
        year: int | None
        __extra_items__: int | None

    details: MovieWithYear = {"name": "Kill Bill Vol. 1", "year": 2003}
    movie: Movie = details  # Not OK. 'year' is not required in 'Movie',
                            # so it shouldn't be required in 'MovieWithYear' either

Because "year" is absent in Movie, __extra_items__ is considered the
corresponding key. "year" being required violates the rule "For each
required key in B, the corresponding key is required in A".

When __extra_items__ is defined to be read-only in a TypedDict type, it
is possible for an item to have a narrower type than __extra_items__'s
value type:

    class Movie(TypedDict, closed=True):
        name: str
        __extra_items__: ReadOnly[str | int]

    class MovieDetails(TypedDict, closed=True):
        name: str
        year: NotRequired[int]
        __extra_items__: int

    details: MovieDetails = {"name": "Kill Bill Vol. 2", "year": 2004}
    movie: Movie = details  # OK. 'int' is consistent with 'str | int'.

This behaves the same way as PEP 705 specified if
year: ReadOnly[str | int] is an item defined in Movie.

__extra_items__ as a pseudo-item follows the same rules that other items
have, so when both TypedDicts contain __extra_items__, this check is
naturally enforced:

    class MovieExtraInt(TypedDict, closed=True):
        name: str
        __extra_items__: int

    class MovieExtraStr(TypedDict, closed=True):
        name: str
        __extra_items__: str

    extra_int: MovieExtraInt = {"name": "No Country for Old Men", "year": 2007}
    extra_str: MovieExtraStr = {"name": "No Country for Old Men", "description": ""}
    extra_int = extra_str  # Not OK. 'str' is inconsistent with 'int' for item '__extra_items__'
    extra_str = extra_int  # Not OK. 'int' is inconsistent with 'str' for item '__extra_items__'

A non-closed TypedDict type implicitly allows non-required extra keys of
value type ReadOnly[object]. This allows to apply the type consistency
rules between this type and a closed TypedDict type:

    class MovieNotClosed(TypedDict):
        name: str

    extra_int: MovieExtraInt = {"name": "No Country for Old Men", "year": 2007}
    not_closed: MovieNotClosed = {"name": "No Country for Old Men"}
    extra_int = not_closed  # Not OK. 'ReadOnly[object]' implicitly on 'MovieNotClosed' is not consistent with 'int' for item '__extra_items__'
    not_closed = extra_int  # OK

Interaction with Constructors

TypedDicts that allow extra items of type T also allow arbitrary keyword
arguments of this type when constructed by calling the class object:

    class OpenMovie(TypedDict):
        name: str

    OpenMovie(name="No Country for Old Men")  # OK
    OpenMovie(name="No Country for Old Men", year=2007)  # Not OK. Unrecognized key

    class ExtraMovie(TypedDict, closed=True):
        name: str
        __extra_items__: int

    ExtraMovie(name="No Country for Old Men")  # OK
    ExtraMovie(name="No Country for Old Men", year=2007)  # OK
    ExtraMovie(
        name="No Country for Old Men",
        language="English",
    )  # Not OK. Wrong type for extra key

    # This implies '__extra_items__: Never',
    # so extra keyword arguments produce an error
    class ClosedMovie(TypedDict, closed=True):
        name: str

    ClosedMovie(name="No Country for Old Men")  # OK
    ClosedMovie(
        name="No Country for Old Men",
        year=2007,
    )  # Not OK. Extra items not allowed

Interaction with Mapping[KT, VT]

A TypedDict type can be consistent with Mapping[KT, VT] types other than
Mapping[str, object] as long as the union of value types on the
TypedDict type is consistent with VT. It is an extension of this rule
from the typing spec:

  -   A TypedDict with all int values is not consistent with
      Mapping[str, int], since there may be additional non-int values
      not visible through the type, due to structural subtyping. These
      can be accessed using the values() and items() methods in Mapping

For example:

    class MovieExtraStr(TypedDict, closed=True):
        name: str
        __extra_items__: str

    extra_str: MovieExtraStr = {"name": "Blade Runner", "summary": ""}
    str_mapping: Mapping[str, str] = extra_str  # OK

    int_mapping: Mapping[str, int] = extra_int  # Not OK. 'int | str' is not consistent with 'int'
    int_str_mapping: Mapping[str, int | str] = extra_int  # OK

Furthermore, type checkers should be able to infer the precise return
types of values() and items() on such TypedDict types:

    def fun(movie: MovieExtraStr) -> None:
        reveal_type(movie.items())  # Revealed type is 'dict_items[str, str]'
        reveal_type(movie.values())  # Revealed type is 'dict_values[str, str]'

Interaction with dict[KT, VT]

Note that because the presence of __extra_items__ on a closed TypedDict
type prohibits additional required keys in its structural subtypes, we
can determine if the TypedDict type and its structural subtypes will
ever have any required key during static analysis.

The TypedDict type is consistent with dict[str, VT] if all items on the
TypedDict type satisfy the following conditions:

-   VT is consistent with the value type of the item
-   The value type of the item is consistent with VT
-   The item is not read-only.
-   The item is not required.

For example:

    class IntDict(TypedDict, closed=True):
        __extra_items__: int

    class IntDictWithNum(IntDict):
        num: NotRequired[int]

    def f(x: IntDict) -> None:
        v: dict[str, int] = x  # OK
        v.clear()  # OK

    not_required_num: IntDictWithNum = {"num": 1, "bar": 2} 
    regular_dict: dict[str, int] = not_required_num  # OK
    f(not_required_num)  # OK

In this case, methods that are previously unavailable on a TypedDict are
allowed:

    not_required_num.clear()  # OK

    reveal_type(not_required_num.popitem())  # OK. Revealed type is tuple[str, int]

However, dict[str, VT] is not necessarily consistent with a TypedDict
type, because such dict can be a subtype of dict:

    class CustomDict(dict[str, int]):
        ...

    not_a_regular_dict: CustomDict = {"num": 1}
    int_dict: IntDict = not_a_regular_dict  # Not OK

How to Teach This

The choice of the spelling "__extra_items__" is intended to make this
feature more understandable to new users compared to shorter
alternatives like "__extra__".

Details of this should be documented in both the typing spec and the
typing documentation.

Backwards Compatibility

Because __extra_items__ remains as a regular key if closed=True is not
specified, no existing codebase will break due to this change.

If the proposal is accepted, none of __required_keys__,
__optional_keys__, __readonly_keys__ and __mutable_keys__ should include
"__extra_items__" defined on the same TypedDict type when closed=True is
specified.

Note that closed as a keyword argument does not collide with the keyword
arguments alternative to define keys with the functional syntax that
allows things like TD = TypedDict("TD", foo=str, bar=int), because it is
scheduled to be removed in Python 3.13.

Because this is a type-checking feature, it can be made available to
older versions as long as the type checker supports it.

Rejected Ideas

Allowing Extra Items without Specifying the Type

extra=True was originally proposed for defining a TypedDict that accepts
extra items regardless of the type, like how total=True works:

    class TypedDict(extra=True):
        pass

Because it did not offer a way to specify the type of the extra items,
the type checkers will need to assume that the type of the extra items
is Any, which compromises type safety. Furthermore, the current behavior
of TypedDict already allows untyped extra items to be present in
runtime, due to structural subtyping. closed=True plays a similar role
in the current proposal.

Supporting TypedDict(extra=type)

During the discussion of the PEP, there were strong objections against
adding another place where types are passed as values instead of
annotations from some authors of type checkers. While this design is
potentially viable, there are also several partially addressable
concerns to consider.

-   Usability of forward reference As in the functional syntax, using a
    quoted type or a type alias will be required when SomeType is a
    forward reference. This is already a requirement for the functional
    syntax, so implementations can potentially reuse that piece of
    logic, but this is still extra work that the closed=True proposal
    doesn't have.
-   Concerns about using type as a value Whatever is not allowed as the
    value type in the functional syntax should not be allowed as the
    argument for extra either. While type checkers might be able to
    reuse this check, it still needs to be somehow special-cased for the
    class-based syntax.
-   How to teach Notably, the extra=type often gets brought up due to it
    being an intuitive solution for the use case, so it is potentially
    simpler to learn than the less obvious solution. However, the more
    common used case only requires closed=True, and the other drawbacks
    mentioned earlier outweigh what is need to teach the usage of the
    special key.

Support Extra Items with Intersection

Supporting intersections in Python's type system requires a lot of
careful consideration, and it can take a long time for the community to
reach a consensus on a reasonable design.

Ideally, extra items in TypedDict should not be blocked by work on
intersections, nor does it necessarily need to be supported through
intersections.

Moreover, the intersection between Mapping[...] and TypedDict is not
equivalent to a TypedDict type with the proposed __extra_items__ special
item, as the value type of all known items in TypedDict needs to satisfy
the is-subtype-of relation with the value type of Mapping[...].

Requiring Type Compatibility of the Known Items with __extra_items__

__extra_items__ restricts the value type for keys that are unknown to
the TypedDict type. So the value type of any known item is not
necessarily consistent with __extra_items__'s type, and
__extra_items__'s type is not necessarily consistent with the value
types of all known items.

This differs from TypeScript's Index Signatures syntax, which requires
all properties' types to match the string index's type. For example:

    interface MovieWithExtraNumber {
        name: string // Property 'name' of type 'string' is not assignable to 'string' index type 'number'.
        [index: string]: number
    }

    interface MovieWithExtraNumberOrString {
        name: string // OK
        [index: string]: number | string
    }

This is a known limitation discussed in TypeScript's issue tracker,
where it is suggested that there should be a way to exclude the defined
keys from the index signature so that it is possible to define a type
like MovieWithExtraNumber.

Reference Implementation

This proposal is supported in pyright 1.1.352, and pyanalyze 0.12.0.

Acknowledgments

Thanks to Jelle Zijlstra for sponsoring this PEP and providing review
feedback, Eric Traut who proposed the original design this PEP iterates
on, and Alice Purcell for offering their perspective as the author of
PEP 705.

Copyright

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.