PEP: 705 Title: TypedDict: Read-only items Author: Alice Purcell
<alicederyn@gmail.com> Sponsor: Pablo Galindo <pablogsal@gmail.com>
Discussions-To:
https://discuss.python.org/t/pep-705-read-only-typeddict-items/37867
Status: Final Type: Standards Track Topic: Typing Created: 07-Nov-2022
Python-Version: 3.13 Post-History: 30-Sep-2022, 02-Nov-2022,
14-Mar-2023, 17-Oct-2023, 04-Nov-2023, Resolution: 29-Feb-2024

typing:readonly and typing.ReadOnly

Abstract

PEP 589 defines the structural type ~typing.TypedDict for dictionaries
with a fixed set of keys. As TypedDict is a mutable type, it is
difficult to correctly annotate methods which accept read-only
parameters in a way that doesn't prevent valid inputs.

This PEP proposes a new type qualifier, typing.ReadOnly, to support
these usages. It makes no Python grammar changes. Correct usage of
read-only keys of TypedDicts is intended to be enforced only by static
type checkers, and will not be enforced by Python itself at runtime.

Motivation

Representing structured data using (potentially nested) dictionaries
with string keys is a common pattern in Python programs. PEP 589 allows
these values to be type checked when the exact type is known up-front,
but it is hard to write read-only code that accepts more specific
variants: for instance, where values may be subtypes or restrict a union
of possible types. This is an especially common issue when writing APIs
for services, which may support a wide range of input structures, and
typically do not need to modify their input.

Pure functions

Consider trying to add type hints to a function movie_string:

    def movie_string(movie: Movie) -> str:
        if movie.get("year") is None:
            return movie["name"]
        else:
            return f'{movie["name"]} ({movie["year"]})'

We could define this Movie type using a TypedDict:

    from typing import NotRequired, TypedDict

    class Movie(TypedDict):
        name: str
        year: NotRequired[int | None]

But suppose we have another type where year is required:

    class MovieRecord(TypedDict):
        name: str
        year: int

Attempting to pass a MovieRecord into movie_string results in the error
(using mypy):

    Argument 1 to "movie_string" has incompatible type "MovieRecord"; expected "Movie"

This particular use case should be type-safe, but the type checker
correctly stops the user from passing a MovieRecord into a Movie
parameter in the general case, because the Movie class has mutator
methods that could potentially allow the function to break the type
constraints in MovieRecord (e.g. with movie["year"] = None or
del movie["year"]). The problem disappears if we don't have mutator
methods in Movie. This could be achieved by defining an immutable
interface using a PEP 544 ~typing.Protocol:

    from typing import Literal, Protocol, overload

    class Movie(Protocol):
        @overload
        def get(self, key: Literal["name"]) -> str: ...

        @overload
        def get(self, key: Literal["year"]) -> int | None: ...

        @overload
        def __getitem__(self, key: Literal["name"]) -> str: ...

        @overload
        def __getitem__(self, key: Literal["year"]) -> int | None: ...

This is very repetitive, easy to get wrong, and is still missing
important method definitions like __contains__() and keys().

Updating nested dicts

The structural typing of TypedDict is supposed to permit writing update
functions that only constrain the types of items they modify:

    class HasTimestamp(TypedDict):
        timestamp: float

    class Logs(TypedDict):
        timestamp: float
        loglines: list[str]

    def update_timestamp(d: HasTimestamp) -> None:
        d["timestamp"] = now()

    def add_logline(logs: Logs, logline: str) -> None:
        logs["loglines"].append(logline)
        update_timestamp(logs)  # Accepted by type checker

However, this no longer works once you start nesting dictionaries:

    class HasTimestampedMetadata(TypedDict):
        metadata: HasTimestamp

    class UserAudit(TypedDict):
        name: str
        metadata: Logs

    def update_metadata_timestamp(d: HasTimestampedMetadata) -> None:
        d["metadata"]["timestamp"] = now()

    def rename_user(d: UserAudit, name: str) -> None:
        d["name"] = name
        update_metadata_timestamp(d)  # Type check error: "metadata" is not of type HasTimestamp

This looks like an error, but is simply due to the (unwanted) ability to
overwrite the metadata item held by the HasTimestampedMetadata instance
with a different HasTimestamp instance, that may no longer be a Logs
instance.

It is possible to work around this issue with generics (as of Python
3.11), but it is very complicated, requiring a type parameter for every
nested dict.

Rationale

These problems can be resolved by removing the ability to update one or
more of the items in a TypedDict. This does not mean the items are
immutable: a reference to the underlying dictionary could still exist
with a different but compatible type in which those items have mutator
operations. These items are "read-only", and we introduce a new
typing.ReadOnly type qualifier for this purpose.

The movie_string function in the first motivating example can then be
typed as follows:

    from typing import NotRequired, ReadOnly, TypedDict

    class Movie(TypedDict):
        name: ReadOnly[str]
        year: ReadOnly[NotRequired[int | None]]

    def movie_string(movie: Movie) -> str:
        if movie.get("year") is None:
            return movie["name"]
        else:
            return f'{movie["name"]} ({movie["year"]})'

A mixture of read-only and non-read-only items is permitted, allowing
the second motivating example to be correctly annotated:

    class HasTimestamp(TypedDict):
        timestamp: float

    class HasTimestampedMetadata(TypedDict):
        metadata: ReadOnly[HasTimestamp]

    def update_metadata_timestamp(d: HasTimestampedMetadata) -> None:
        d["metadata"]["timestamp"] = now()

    class Logs(HasTimestamp):
        loglines: list[str]

    class UserAudit(TypedDict):
        name: str
        metadata: Logs

    def rename_user(d: UserAudit, name: str) -> None:
        d["name"] = name
        update_metadata_timestamp(d)  # Now OK

In addition to these benefits, by flagging arguments of a function as
read-only (by using a TypedDict like Movie with read-only items), it
makes explicit not just to typecheckers but also to users that the
function is not going to modify its inputs, which is usually a desirable
property of a function interface.

This PEP proposes making ReadOnly valid only in a TypedDict. A possible
future extension would be to support it in additional contexts, such as
in protocols.

Specification

A new typing.ReadOnly type qualifier is added.

typing.ReadOnly type qualifier

The typing.ReadOnly type qualifier is used to indicate that an item
declared in a TypedDict definition may not be mutated (added, modified,
or removed):

    from typing import ReadOnly

    class Band(TypedDict):
        name: str
        members: ReadOnly[list[str]]

    blur: Band = {"name": "blur", "members": []}
    blur["name"] = "Blur"  # OK: "name" is not read-only
    blur["members"] = ["Damon Albarn"]  # Type check error: "members" is read-only
    blur["members"].append("Damon Albarn")  # OK: list is mutable

Alternative functional syntax

The alternative functional syntax <589#alternative-syntax> for TypedDict
also supports the new type qualifier:

    Band = TypedDict("Band", {"name": str, "members": ReadOnly[list[str]]})

Interaction with other special types

ReadOnly[] can be used with Required[], NotRequired[] and Annotated[],
in any nesting order:

    class Movie(TypedDict):
        title: ReadOnly[Required[str]]  # OK
        year: ReadOnly[NotRequired[Annotated[int, ValueRange(-9999, 9999)]]]  # OK

    class Movie(TypedDict):
        title: Required[ReadOnly[str]]  # OK
        year: Annotated[NotRequired[ReadOnly[int]], ValueRange(-9999, 9999)]  # OK

This is consistent with the behavior introduced in PEP 655.

Inheritance

Subclasses can redeclare read-only items as non-read-only, allowing them
to be mutated:

    class NamedDict(TypedDict):
        name: ReadOnly[str]

    class Album(NamedDict):
        name: str
        year: int

    album: Album = { "name": "Flood", "year": 1990 }
    album["year"] = 1973
    album["name"] = "Dark Side Of The Moon"  # OK: "name" is not read-only in Album

If a read-only item is not redeclared, it remains read-only:

    class Album(NamedDict):
        year: int

    album: Album = { "name": "Flood", "year": 1990 }
    album["name"] = "Dark Side Of The Moon"  # Type check error: "name" is read-only in Album

Subclasses can narrow value types of read-only items:

    class AlbumCollection(TypedDict):
        albums: ReadOnly[Collection[Album]]

    class RecordShop(AlbumCollection):
        name: str
        albums: ReadOnly[list[Album]]  # OK: "albums" is read-only in AlbumCollection

Subclasses can require items that are read-only but not required in the
superclass:

    class OptionalName(TypedDict):
        name: ReadOnly[NotRequired[str]]

    class RequiredName(OptionalName):
        name: ReadOnly[Required[str]]

    d: RequiredName = {}  # Type check error: "name" required

Subclasses can combine these rules:

    class OptionalIdent(TypedDict):
        ident: ReadOnly[NotRequired[str | int]]

    class User(OptionalIdent):
        ident: str  # Required, mutable, and not an int

Note that these are just consequences of structural typing, but they are
highlighted here as the behavior now differs from the rules specified in
PEP 589.

Type consistency

This section updates the type consistency rules introduced in PEP 589 to
cover the new feature in this PEP. In particular, any pair of types that
do not use the new feature will be consistent under these new rules if
(and only if) they were already consistent.

A TypedDict type A is consistent with TypedDict B if A is structurally
compatible with B. This is true if and only if all of the following are
satisfied:

-   For each item in B, A has the corresponding key, unless the item in
    B is read-only, not required, and of top value type
    (ReadOnly[NotRequired[object]]).
-   For each item in B, if A has the corresponding key, the
    corresponding value type in A is consistent with the value type in
    B.
-   For each non-read-only item in B, its value type is consistent with
    the corresponding value type in A.
-   For each required key in B, the corresponding key is required in A.
-   For each non-required key in B, if the item is not read-only in B,
    the corresponding key is not required in A.

Discussion:

-   All non-specified items in a TypedDict implicitly have value type
    ReadOnly[NotRequired[object]].

-   Read-only items behave covariantly, as they cannot be mutated. This
    is similar to container types such as Sequence, and different from
    non-read-only items, which behave invariantly. Example:

        class A(TypedDict):
            x: ReadOnly[int | None]

        class B(TypedDict):
            x: int

        def f(a: A) -> None:
            print(a["x"] or 0)

        b: B = {"x": 1}
        f(b)  # Accepted by type checker

-   A TypedDict type A with no explicit key 'x' is not consistent with a
    TypedDict type B with a non-required key 'x', since at runtime the
    key 'x' could be present and have an incompatible type (which may
    not be visible through A due to structural subtyping). The only
    exception to this rule is if the item in B is read-only, and the
    value type is of top type (object). For example:

        class A(TypedDict):
            x: int

        class B(TypedDict):
            x: int
            y: ReadOnly[NotRequired[object]]

        a: A = { "x": 1 }
        b: B = a  # Accepted by type checker

Update method

In addition to existing type checking rules, type checkers should error
if a TypedDict with a read-only item is updated with another TypedDict
that declares that key:

    class A(TypedDict):
        x: ReadOnly[int]
        y: int

    a1: A = { "x": 1, "y": 2 }
    a2: A = { "x": 3, "y": 4 }
    a1.update(a2)  # Type check error: "x" is read-only in A

Unless the declared value is of bottom type (~typing.Never):

    class B(TypedDict):
        x: NotRequired[typing.Never]
        y: ReadOnly[int]

    def update_a(a: A, b: B) -> None:
        a.update(b)  # Accepted by type checker: "x" cannot be set on b

Note: Nothing will ever match the Never type, so an item annotated with
it must be absent.

Keyword argument typing

PEP 692 introduced Unpack to annotate **kwargs with a TypedDict. Marking
one or more of the items of a TypedDict used in this way as read-only
will have no effect on the type signature of the method. However, it
will prevent the item from being modified in the body of the function:

    class Args(TypedDict):
        key1: int
        key2: str

    class ReadOnlyArgs(TypedDict):
        key1: ReadOnly[int]
        key2: ReadOnly[str]

    class Function(Protocol):
        def __call__(self, **kwargs: Unpack[Args]) -> None: ...

    def impl(**kwargs: Unpack[ReadOnlyArgs]) -> None:
        kwargs["key1"] = 3  # Type check error: key1 is readonly

    fn: Function = impl  # Accepted by type checker: function signatures are identical

Runtime behavior

TypedDict types will gain two new attributes, __readonly_keys__ and
__mutable_keys__, which will be frozensets containing all read-only and
non-read-only keys, respectively:

    class Example(TypedDict):
        a: int
        b: ReadOnly[int]
        c: int
        d: ReadOnly[int]

    assert Example.__readonly_keys__ == frozenset({'b', 'd'})
    assert Example.__mutable_keys__ == frozenset({'a', 'c'})

typing.get_type_hints will strip out any ReadOnly type qualifiers,
unless include_extras is True:

    assert get_type_hints(Example)['b'] == int
    assert get_type_hints(Example, include_extras=True)['b'] == ReadOnly[int]

typing.get_origin and typing.get_args will be updated to recognize
ReadOnly:

    assert get_origin(ReadOnly[int]) is ReadOnly
    assert get_args(ReadOnly[int]) == (int,)

Backwards compatibility

This PEP adds a new feature to TypedDict, so code that inspects
TypedDict types will have to change to support types using it. This is
expected to mainly affect type-checkers.

Security implications

There are no known security consequences arising from this PEP.

How to teach this

Suggested changes to the typing module documentation, in line with
current practice:

-   Add this PEP to the others listed.
-   Add typing.ReadOnly, linked to TypedDict and this PEP.
-   Add the following text to the TypedDict entry:

The ReadOnly type qualifier indicates that an item declared in a
TypedDict definition may be read but not mutated (added, modified or
removed). This is useful when the exact type of the value is not known
yet, and so modifying it would break structural subtypes. insert example

Reference implementation

pyright 1.1.333 fully implements this proposal.

Rejected alternatives

A TypedMapping protocol type

An earlier version of this PEP proposed a TypedMapping protocol type,
behaving much like a read-only TypedDict but without the constraint that
the runtime type be a dict. The behavior described in the current
version of this PEP could then be obtained by inheriting a TypedDict
from a TypedMapping. This has been set aside for now as more complex,
without a strong use-case motivating the additional complexity.

A higher-order ReadOnly type

A generalized higher-order type could be added that removes mutator
methods from its parameter, e.g. ReadOnly[MovieRecord]. For a TypedDict,
this would be like adding ReadOnly to every item, including those
declared in superclasses. This would naturally want to be defined for a
wider set of types than just TypedDict subclasses, and also raises
questions about whether and how it applies to nested types. We decided
to keep the scope of this PEP narrower.

Calling the type Readonly

Read-only is generally hyphenated, and it appears to be common
convention to put initial caps onto words separated by a dash when
converting to CamelCase. This appears consistent with the definition of
CamelCase on Wikipedia: CamelCase uppercases the first letter of each
word. That said, Python examples or counter-examples, ideally from the
core Python libraries, or better explicit guidance on the convention,
would be greatly appreciated.

Reusing the Final annotation

The ~typing.Final annotation prevents an attribute from being modified,
like the proposed ReadOnly qualifier does for TypedDict items. However,
it is also documented as preventing redefinition in subclasses too; from
PEP 591:

  The typing.Final type qualifier is used to indicate that a variable or
  attribute should not be reassigned, redefined, or overridden.

This does not fit with the intended use of ReadOnly. Rather than
introduce confusion by having Final behave differently in different
contexts, we chose to introduce a new qualifier.

A readonly flag

Earlier versions of this PEP introduced a boolean flag that would ensure
all items in a TypedDict were read-only:

    class Movie(TypedDict, readonly=True):
        name: str
        year: NotRequired[int | None]

    movie: Movie = { "name": "A Clockwork Orange" }
    movie["year"] = 1971  # Type check error: "year" is read-only

However, this led to confusion when inheritance was introduced:

    class A(TypedDict):
        key1: int

    class B(A, TypedDict, readonly=True):
        key2: int

    b: B = { "key1": 1, "key2": 2 }
    b["key1"] = 4  # Accepted by type checker: "key1" is not read-only

It would be reasonable for someone familiar with frozen (from
dataclasses), on seeing just the definition of B, to assume that the
whole type was read-only. On the other hand, it would be reasonable for
someone familiar with total to assume that read-only only applies to the
current type.

The original proposal attempted to eliminate this ambiguity by making it
both a type check and a runtime error to define B in this way. This was
still a source of surprise to people expecting it to work like total.

Given that no extra types could be expressed with the readonly flag, it
has been removed from the proposal to avoid ambiguity and surprise.

Supporting type-checked removal of read-only qualifier via copy and other methods

An earlier version of this PEP mandated that code like the following be
supported by type-checkers:

    class A(TypedDict):
        x: ReadOnly[int]

    class B(TypedDict):
        x: ReadOnly[str]

    class C(TypedDict):
        x: int | str

    def copy_and_modify(a: A) -> C:
        c: C = copy.copy(a)
        if not c['x']:
            c['x'] = "N/A"
        return c

    def merge_and_modify(a: A, b: B) -> C:
        c: C = a | b
        if not c['x']:
            c['x'] = "N/A"
        return c

However, there is currently no way to express this in the typeshed,
meaning type-checkers would be forced to special-case these functions.
There is already a way to code these operations that mypy and pyright do
support, though arguably this is less readable:

    copied: C = { **a }
    merged: C = { **a, **b }

While not as flexible as would be ideal, the current typeshed stubs are
sound, and remain so if this PEP is accepted. Updating the typeshed
would require new typing features, like a type constructor to express
the type resulting from merging two or more dicts, and a type qualifier
to indicate a returned value is not shared (so may have type constraints
like read-only and invariance of generics loosened in specific ways),
plus details of how type-checkers would be expected to interpret these
features. These could be valuable additions to the language, but are
outside the scope of this PEP.

Given this, we have deferred any update of the typeshed stubs.

Preventing unspecified keys in TypedDicts

Consider the following "type discrimination" code:

    class A(TypedDict):
      foo: int

    class B(TypedDict):
      bar: int

    def get_field(d: A | B) -> int:
      if "foo" in d:
        return d["foo"]  # !!!
      else:
        return d["bar"]

This is a common idiom, and other languages like Typescript allow it.
Technically, however, this code is unsound: B does not declare foo, but
instances of B may still have the key present, and the associated value
may be of any type:

    class C(TypedDict):
      foo: str
      bar: int

    c: C = { "foo": "hi", "bar" 3 }
    b: B = c  # OK: C is structurally compatible with B
    v = get_field(b)  # Returns a string at runtime, not an int!

mypy rejects the definition of get_field on the marked line with the
error TypedDict "B" has no key "foo", which is a rather confusing error
message, but is caused by this unsoundness.

One option for correcting this would be to explicitly prevent B from
holding a foo:

    class B(TypedDict):
      foo: NotRequired[Never]
      bar: int

    b: B = c  # Type check error: key "foo" not allowed in B

However, this requires every possible key that might be used to
discriminate on to be explicitly declared in every type, which is not
generally feasible. A better option would be to have a way of preventing
all unspecified keys from being included in B. mypy supports this using
the @final decorator from PEP 591:

    @final
    class B(TypedDict):
      bar: int

The reasoning here is that this prevents C or any other type from being
considered a "subclass" of B, so instances of B can now be relied on to
never hold the key foo, even though it is not explicitly declared to be
of bottom type.

With the introduction of read-only items, however, this reasoning would
imply type-checkers should ban the following:

    @final
    class D(TypedDict):
      field: ReadOnly[Collection[str]]

    @final
    class E(TypedDict):
      field: list[str]

    e: E = { "field": ["value1", "value2"] }
    d: D = e  # Error?

The conceptual problem here is that TypedDicts are structural types:
they cannot really be subclassed. As such, using @final on them is not
well-defined; it is certainly not mentioned in PEP 591.

An earlier version of this PEP proposed resolving this by adding a new
flag to TypedDict that would explicitly prevent other keys from being
used, but not other kinds of structural compatibility:

    class B(TypedDict, other_keys=Never):
      bar: int

    b: B = c  # Type check error: key "foo" not allowed in B

However, during the process of drafting, the situation changed:

-   pyright, which previously worked similarly to mypy in this type
    discrimination case, changed to allow the original example without
    error, despite the unsoundness, due to it being a common idiom
-   mypy has an open issue to follow the lead of pyright and Typescript
    and permit the idiom as well
-   a draft of PEP-728 was created that is a superset of the other_keys
    functionality

As such, there is less urgency to address this issue in this PEP, and it
has been deferred to PEP-728.

Copyright

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.