PEP: 746 Title: Type checking Annotated metadata Author: Adrian Garcia
Badaracco <adrian@adriangb.com> Sponsor: Jelle Zijlstra
<jelle.zijlstra@gmail.com> Discussions-To:
https://discuss.python.org/t/pep-746-typedmetadata-for-type-checking-of-pep-593-annotated/53834
Status: Draft Type: Standards Track Topic: Typing Created: 20-May-2024
Python-Version: 3.14 Post-History: 20-May-2024

Abstract

This PEP proposes a mechanism for type checking metadata that uses the
typing.Annotated type. Metadata objects that implement the new
__supports_annotated_base__ protocol will be type checked by static type
checkers to ensure that the metadata is valid for the given type.

Motivation

PEP 593 introduced Annotated as a way to attach runtime metadata to
types. In general, the metadata is not meant for static type checkers,
but even so, it is often useful to be able to check that the metadata
makes sense for the given type.

Take the first example in PEP 593, which uses Annotated to attach
serialization information to a field:

    class Student(struct2.Packed):
        name: Annotated[str, struct2.ctype("<10s")]

Here, the struct2.ctype("<10s") metadata is meant to be used by a
serialization library to serialize the field. Such libraries can only
serialize a subset of types: it would not make sense to write, for
example, Annotated[list[str], struct2.ctype("<10s")]. Yet the type
system provides no way to enforce this. The metadata are completely
ignored by type checkers.

This use case comes up in libraries like pydantic and msgspec, which use
Annotated to attach validation and conversion information to fields or
fastapi, which uses Annotated to mark parameters as extracted from
headers, query strings or dependency injection.

Specification

This PEP introduces a protocol that can be used by static and runtime
type checkers to validate the consistency between Annotated metadata and
a given type. Objects that implement this protocol have an attribute
called __supports_annotated_base__ that specifies whether the metadata
is valid for a given type:

    class Int64:
        __supports_annotated_base__: int

The attribute may also be marked as a ClassVar to avoid interaction with
dataclasses:

    from dataclasses import dataclass
    from typing import ClassVar

    @dataclass
    class Gt:
        value: int
        __supports_annotated_base__: ClassVar[int]

When a static type checker encounters a type expression of the form
Annotated[T, M1, M2, ...], it should enforce that for each metadata
element in M1, M2, ..., one of the following is true:

-   The metadata element evaluates to an object that does not have a
    __supports_annotated_base__ attribute; or
-   The metadata element evaluates to an object M that has a
    __supports_annotated_base__ attribute; and T is assignable to the
    type of M.__supports_annotated_base__.

To support generic Gt metadata, one might write:

    from typing import Protocol

    class SupportsGt[T](Protocol):
        def __gt__(self, __other: T) -> bool:
            ...

    class Gt[T]:
        __supports_annotated_base__: ClassVar[SupportsGt[T]]

        def __init__(self, value: T) -> None:
            self.value = value

    x1: Annotated[int, Gt(0)] = 1  # OK
    x2: Annotated[str, Gt(0)] = 0  # type checker error: str is not assignable to SupportsGt[int]
    x3: Annotated[int, Gt(1)] = 0  # OK for static type checkers; runtime type checkers may flag this

Backwards Compatibility

Metadata that does not implement the protocol will be considered valid
for all types, so no breaking changes are introduced for existing code.
The new checks only apply to metadata objects that explicitly implement
the protocol specified by this PEP.

Security Implications

None.

How to Teach This

This protocol is intended mostly for libraries that provide Annotated
metadata; end users of those libraries are unlikely to need to implement
the protocol themselves. The protocol should be mentioned in the
documentation for typing.Annotated and in the typing specification.

Reference Implementation

None yet.

Rejected ideas

Introducing a type variable instead of a generic class

We considered using a special type variable,
AnnotatedT = TypeVar("AnnotatedT"), to represent the type T of the inner
type in Annotated; metadata would be type checked against this type
variable. However, this would require using the old type variable syntax
(before PEP 695), which is now a discouraged feature. In addition, this
would use type variables in an unusual way that does not fit well with
the rest of the type system.

Introducing a new type to typing.py that all metadata objects should subclass

A previous version of this PEP suggested adding a new generic base
class, TypedMetadata[U], that metadata objects would subclass. If a
metadata object is a subclass of TypedMetadata[U], then type checkers
would check that the annotation's base type is assignable to U. However,
this mechanism does not integrate as well with the rest of the language;
Python does not generally use marker base classes. In addition, it
provides less flexibility than the current proposal: it would not allow
overloads, and it would require metadata objects to add a new base
class, which may make their runtime implementation more complex.

Using a method instead of an attribute for __supports_annotated_base__

We considered using a method instead of an attribute for the protocol,
so that this method can be used at runtime to check the validity of the
metadata and to support overloads or returning boolean literals.
However, using a method adds boilerplate to the implementation and the
value of the runtime use cases or more complex scenarios involving
overloads and returning boolean literals was not clear.

Acknowledgments

We thank Eric Traut for suggesting the idea of using a protocol and
implementing provisional support in Pyright. Thank you to Jelle Zijlstra
for sponsoring this PEP.

Copyright

This document has been placed in the public domain.