PEP: 743 Title: Add Py_COMPAT_API_VERSION to the Python C API Author:
Victor Stinner <vstinner@python.org>, Petr Viktorin <encukou@gmail.com>,
PEP-Delegate: C API Working Group Status: Draft Type: Standards Track
Created: 11-Mar-2024 Python-Version: 3.14

Abstract

Add Py_COMPAT_API_VERSION C macro that hides some deprecated and
soft-deprecated symbols, allowing users to opt out of using API with
known issues that other API solves. The macro is versioned, allowing
users to update (or not) on their own pace.

Also, add namespaced alternatives for API without the Py_ prefix, and
soft-deprecate the original names.

Motivation

Some of Python's C API has flaws that are only obvious in hindsight.

If an API prevents adding features or optimizations, or presents a
serious security risk or maintenance burden, we can deprecate and remove
it as described in PEP 387.

However, this leaves us with some API that has “sharp edges” -- it works
fine for its current users, but should be avoided in new code. For
example:

-   API that cannot signal an exception, so failures are either ignored
    or exit the process with a fatal error. For example
    PyObject_HasAttr.
-   API that is not thread-safe, for example by borrowing references
    from mutable objects, or exposing unfinished mutable objects. For
    example PyDict_GetItemWithError.
-   API with names that don't use the Py/_Py prefix, and so can clash
    with other code. For example: setter.

It is important to note that despite such flaws, it's usually possible
to use the API correctly. For example, in a single-threaded environment,
thread safety is not an issue. We do not want to break working code,
even if it uses API that would be wrong in some -- or even most -- other
contexts.

On the other hand, we want to steer users away from such “undesirable”
API in new code, especially if a safer alternative exists.

Adding the Py prefix

Some names defined in CPython headers is not namespaced: it that lacks
the Py prefix (or a variant: _Py, and alternative capitalizations). For
example, we declare a function type named simply setter.

While such names are not exported in the ABI (as checked by
make smelly), they can clash with user code and, more importantly, with
libraries linked to third-party extensions.

While it would be possible to provide namespaced aliases and
(soft-)deprecate these names, the only way to make them not clash with
third-party code is to not define them in Python headers at all.

Rationale

We want to allow an easy way for users to avoid “undesirable” API if
they choose to do so.

It might be be sufficient to leave this to third-party linters. For that
we'd need a good way to expose a list of (soft-)deprecated API to such
linters. While adding that, we can -- rather easily -- do the linter's
job directly in CPython headers, avoiding the neel for an extra tool.
Unlike Python, C makes it rather easy to limit available API -- for a
whole project or for each individual source file -- by having users
define an “opt-in” macro.

We already do something similar with Py_LIMITED_API, which limits the
available API to a subset that compiles to stable ABI. (In hindsight, we
should have used a different macro name for that particular kind of
limiting, but it's too late to change that now.)

To prevent working code from breaking as we identify more “undesirable”
API and add safer alternatives to it, the opt-in macro should be
versioned. Users can choose a version they need based on their
compatibility requirements, and update it at their own pace.

To be clear, this mechanism is not a replacement for deprecation.
Deprecation is for API that prevents new features or optimizations, or
presents a security risk or maintenance burden. This mechanism, on the
other hand, is meant for cases where “we found a slightly better way of
doing things” -- perhaps one that's harder to misuse, or just has a less
misleading name. (On a lighter note: many people configure a code
quality checker to shout at them about the number of blank lines between
functions. Let's help them identify more substantial “code smells”!)

The proposed macro does not change any API definitions; it only hides
them. So, if code compiles with the macro, it'll also compile without
it, with identical behaviour. This has implications for core devs: to
deal with undesirable behaviour, we'll need to introduce new, better
API, and then discourage the old one. In turn, this implies that we
should look at an individual API and fix all its known issues at once,
rather than do codebase-wide sweeps for a single kind of issue, so that
we avoid multiple renames of the same function.

Adding the Py prefix

An opt-in macro allows us to omit definitions that could clash with
third-party libraries.

Specification

We introduce a Py_COMPAT_API_VERSION macro. If this macro is defined
before #include <Python.h>, some API definitions -- as described below
-- will be omitted from the Python header files.

The macro only omits complete top-level definitions exposed from
<Python.h>. Other things (the ABI, structure definitions, macro
expansions, static inline function bodies, etc.) are not affected.

The C API working group (PEP 731) has authority over the set of omitted
definitions.

The set of omitted definitions will be tied to a particular feature
release of CPython, and is finalized in each 3.x.0 Beta 1 release. In
rare cases, entries can be removed (i.e. made available for use) at any
time.

The macro should be defined to a version in the format used by
PY_VERSION_HEX, with the “micro”, “release” and “serial” fields set to
zero. For example, to omit API deemed undesirable in 3.14.0b1, users
should define Py_COMPAT_API_VERSION to 0x030e0000.

Requirements for omitted API

An API that is omitted with Py_COMPAT_API_VERSION must:

-   be soft-deprecated (see PEP 387);
-   for all known use cases of the API, have a documented alternative or
    workaround;
-   have tests to ensure it keeps working (except for 1:1 renames using
    #define or typedef);
-   be documented (except if it was never mentioned in previous versions
    of the documentation); and
-   be approved by the C API working group. (The WG may give blanket
    approvals for groups of related API; see Initial set below for
    examples.)

Note that Py_COMPAT_API_VERSION is meant for API that can be trivially
replaced by a better alternative. API without a replacement should
generally be deprecated instead.

Location

All API definitions omitted by Py_COMPAT_API_VERSION will be moved to a
new header, Include/legacy.h.

This is meant to help linter authors compile lists, so they can flag the
API with warnings rather than errors.

Note that for simple renaming of source-only constructs (macros, types),
we expect names to be omitted in the same version -- or the same PR --
that adds a replacement. This means that the original definition will be
renamed, and a typedef or #define for the old name added to
Include/legacy.h.

Documentation

Documentation for omitted API should generally:

-   appear after the recommended replacement,
-   reference the replacement (e.g. “Similar to X, but…”), and
-   focus on differences from the replacement and migration advice.

Exceptions are possible if there is a good reason for them.

Initial set

The following API will be omitted with Py_COMPAT_API_VERSION set to
0x030e0000 (3.14) or greater:

-   Omit API returning borrowed references:

      Omitted API              Replacement
      ------------------------ ---------------------------
      PyDict_GetItem()         PyDict_GetItemRef()
      PyDict_GetItemString()   PyDict_GetItemStringRef()
      PyImport_AddModule()     PyImport_AddModuleRef()
      PyList_GetItem()         PyList_GetItemRef()

-   Omit deprecated APIs:

      Omitted Deprecated API             Replacement
      ---------------------------------- -----------------------------------------
      PY_FORMAT_SIZE_T                   "z"
      PY_UNICODE_TYPE                    wchar_t
      PyCode_GetFirstFree()              PyUnstable_Code_GetFirstFree()
      PyCode_New()                       PyUnstable_Code_New()
      PyCode_NewWithPosOnlyArgs()        PyUnstable_Code_NewWithPosOnlyArgs()
      PyImport_ImportModuleNoBlock()     PyImport_ImportModule()
      PyMem_DEL()                        PyMem_Free()
      PyMem_Del()                        PyMem_Free()
      PyMem_FREE()                       PyMem_Free()
      PyMem_MALLOC()                     PyMem_Malloc()
      PyMem_NEW()                        PyMem_New()
      PyMem_REALLOC()                    PyMem_Realloc()
      PyMem_RESIZE()                     PyMem_Resize()
      PyModule_GetFilename()             PyModule_GetFilenameObject()
      PyOS_AfterFork()                   PyOS_AfterFork_Child()
      PyObject_DEL()                     PyObject_Free()
      PyObject_Del()                     PyObject_Free()
      PyObject_FREE()                    PyObject_Free()
      PyObject_MALLOC()                  PyObject_Malloc()
      PyObject_REALLOC()                 PyObject_Realloc()
      PySlice_GetIndicesEx()             (two calls; see current docs)
      PyThread_ReInitTLS()               (no longer needed)
      PyThread_create_key()              PyThread_tss_alloc()
      PyThread_delete_key()              PyThread_tss_free()
      PyThread_delete_key_value()        PyThread_tss_delete()
      PyThread_get_key_value()           PyThread_tss_get()
      PyThread_set_key_value()           PyThread_tss_set()
      PyUnicode_AsDecodedObject()        PyUnicode_Decode()
      PyUnicode_AsDecodedUnicode()       PyUnicode_Decode()
      PyUnicode_AsEncodedObject()        PyUnicode_AsEncodedString()
      PyUnicode_AsEncodedUnicode()       PyUnicode_AsEncodedString()
      PyUnicode_IS_READY()               (no longer needed)
      PyUnicode_READY()                  (no longer needed)
      PyWeakref_GET_OBJECT()             PyWeakref_GetRef()
      PyWeakref_GetObject()              PyWeakref_GetRef()
      Py_UNICODE                         wchar_t
      _PyCode_GetExtra()                 PyUnstable_Code_GetExtra()
      _PyCode_SetExtra()                 PyUnstable_Code_SetExtra()
      _PyDict_GetItemStringWithError()   PyDict_GetItemStringRef()
      _PyEval_RequestCodeExtraIndex()    PyUnstable_Eval_RequestCodeExtraIndex()
      _PyHASH_BITS                       PyHASH_BITS
      _PyHASH_IMAG                       PyHASH_IMAG
      _PyHASH_INF                        PyHASH_INF
      _PyHASH_MODULUS                    PyHASH_MODULUS
      _PyHASH_MULTIPLIER                 PyHASH_MULTIPLIER
      _PyObject_EXTRA_INIT               (no longer needed)
      _PyThreadState_UncheckedGet()      PyThreadState_GetUnchecked()
      _PyUnicode_AsString()              PyUnicode_AsUTF8()
      _Py_HashPointer()                  Py_HashPointer()
      _Py_T_OBJECT                       Py_T_OBJECT_EX
      _Py_WRITE_RESTRICTED               (no longer needed)

-   Soft-deprecate and omit APIs:

      Omitted Deprecated API      Replacement
      --------------------------- -----------------------------------
      PyDict_GetItemWithError()   PyDict_GetItemRef()
      PyDict_SetDefault()         PyDict_SetDefaultRef()
      PyMapping_HasKey()          PyMapping_HasKeyWithError()
      PyMapping_HasKeyString()    PyMapping_HasKeyStringWithError()
      PyObject_HasAttr()          PyObject_HasAttrWithError()
      PyObject_HasAttrString()    PyObject_HasAttrStringWithError()

-   Omit <structmember.h> legacy API:

    The header file structmember.h, which is not included from
    <Python.h> and must be included separately, will #error if
    Py_COMPAT_API_VERSION is defined. This affects the following API:

      Omitted Deprecated API   Replacement
      ------------------------ ---------------------------------
      T_SHORT                  Py_T_SHORT
      T_INT                    Py_T_INT
      T_LONG                   Py_T_LONG
      T_FLOAT                  Py_T_FLOAT
      T_DOUBLE                 Py_T_DOUBLE
      T_STRING                 Py_T_STRING
      T_OBJECT                 (tp_getset; docs to be written)
      T_CHAR                   Py_T_CHAR
      T_BYTE                   Py_T_BYTE
      T_UBYTE                  Py_T_UBYTE
      T_USHORT                 Py_T_USHORT
      T_UINT                   Py_T_UINT
      T_ULONG                  Py_T_ULONG
      T_STRING_INPLACE         Py_T_STRING_INPLACE
      T_BOOL                   Py_T_BOOL
      T_OBJECT_EX              Py_T_OBJECT_EX
      T_LONGLONG               Py_T_LONGLONG
      T_ULONGLONG              Py_T_ULONGLONG
      T_PYSSIZET               Py_T_PYSSIZET
      T_NONE                   (tp_getset; docs to be written)
      READONLY                 Py_READONLY
      PY_AUDIT_READ            Py_AUDIT_READ
      READ_RESTRICTED          Py_AUDIT_READ
      PY_WRITE_RESTRICTED      (no longer needed)
      RESTRICTED               Py_AUDIT_READ

-   Omit soft deprecated macros:

      Omitted Macros     Replacement
      ------------------ -----------------------------
      Py_IS_NAN()        isnan() (C99+ <math.h>)
      Py_IS_INFINITY()   isinf(X) (C99+ <math.h>)
      Py_IS_FINITE()     isfinite(X) (C99+ <math.h>)
      Py_MEMCPY()        memcpy() (C <string.h>)

-   Soft-deprecate and omit typedefs without the Py/_Py prefix (getter,
    setter, allocfunc, …), in favour of new ones that add the prefix
    (Py_getter , etc.)

-   Soft-deprecate and omit macros without the Py/_Py prefix (METH_O,
    CO_COROUTINE, FUTURE_ANNOTATIONS, WAIT_LOCK, …), favour of new ones
    that add the prefix (Py_METH_O , etc.).

-   Any others approved by the C API workgroup

If any of these proposed replacements, or associated documentation, are
not added in time for 3.14.0b1, they'll be omitted with later versions
of Py_COMPAT_API_VERSION. (We expect this for macros generated by
configure: HAVE_*, WITH_*, ALIGNOF_*, SIZEOF_*, and several without a
common prefix.)

Implementation

TBD

Open issues

The name Py_COMPAT_API_VERSION was taken from the earlier PEP; it
doesn't fit this version.

Backwards Compatibility

The macro is backwards compatible. Developers can introduce and update
the macro on their own pace, potentially for one source file at a time.

Discussions

-   C API Evolutions: Macro to hide deprecated functions (October 2023)
-   C API Problems: Opt-in macro for a new clean API? Subset of
    functions with no known issues (June 2023)
-   Finishing the Great Renaming (May 2024)

Prior Art

-   Py_LIMITED_API macro of PEP 384 "Defining a Stable ABI".
-   Rejected PEP 606 "Python Compatibility Version" which has a global
    scope.

Copyright

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.