PEP: 652 Title: Maintaining the Stable ABI Author: Petr Viktorin
<encukou@gmail.com> Discussions-To:
https://discuss.python.org/t/pre-pep-maintaining-the-stable-abi/6986/
Status: Final Type: Standards Track Content-Type: text/x-rst Created:
09-Feb-2021 Python-Version: 3.10 Resolution:
https://mail.python.org/archives/list/python-dev@python.org/message/IN4XMFLQJ6D6V67EXU27GV3QWSEHHNNH/

python:stable (user docs) and devguide:c-api (development docs)

Abstract

CPython's Limited C-API and Stable ABI, introduced in PEP 384, will be
formalized in a single definitive file, tested, and documented.

Motivation

PEP 384 defined a Limited API and Stable ABI, which allow extenders and
embedders of CPython to compile extension modules that are
binary-compatible with any subsequent version of 3.x. In theory, this
brings several advantages:

-   A module can be built only once per platform and support multiple
    versions of Python, reducing time, power and maintainer attention
    needed for builds (in exchange for potentially worse performance).
-   Binary wheels using the Stable ABI work with new versions of CPython
    throughout the pre-release period, and can be tested in environments
    where building from source is not practical.
-   As a welcome side effect of the Limited API's hiding of
    implementation details, this API is becoming a viable target for
    alternate Python implementations that would be incompatible with the
    full C API.

However, in hindsight, PEP 384 and its implementation has several
issues:

-   It is ill-defined. According to PEP 384, functions are opt-out: all
    functions not specially marked are part of the Stable ABI. In
    practice, for Windows there's a list that's opt-in. For users there
    is a #define that should make only the Stable ABI available, but
    there is no process that ensures it is kept up-to date. Neither is
    there a process for updating the documentation.
-   Until recently, the Stable ABI was not tested at all. It tends to
    break. For example, changing a function to a macro can break the
    Stable ABI as the function symbol is removed.
-   There is no way to deprecate parts of the Limited API.
-   It is incomplete. Some operations are not available in the Stable
    ABI, with little reason except "we forgot". (This last point is one
    the PEP will not help with, however.)

This PEP defines the Limited API more clearly and introducess process
designed to make the Stable ABI and Limited API more useful and robust.

Rationale

This PEP contains a lot of clarifications and definitions, but just one
big technical change: the Stable ABI will be explicitly listed in a
human-maintained “manifest” file.

There have been efforts to collect such lists automatically, e.g. by
scanning the symbols exported from Python. Such automation might seem
easier to maintain than a handcrafted file, but has major issues: for
example, the set exported symbols has platform-specific variations.
Also, the cost of updating an explicit manifest is small compared to the
overall work that should go into changing API that will need to be
supported forever (or until Python 3 reaches end of life, if that comes
sooner).

This PEP proposes automatically generating things from the manifest:
initially documentation and DLL contents, with later possibilities for
also automating tests.

Stable ABI vs. Limited API

PEP 384 and this document deal with the Limited API and the Stable ABI,
two related but distinct concepts. In short:

-   The Stable ABI is a promise that certain extensions compiled with
    CPython 3.x will be binary compatible with all subsequent versions
    of CPython 3.x.
-   The Limited API is a subset of CPython's C API that produces such
    extensions.

This section clarifies these terms and defines some of their semantics
(either pre-existing or newly proposed here).

The word “Extensions” is used as a shorthand for all code that uses the
Python API, e.g. extension modules or software that embeds Python.

Stable ABI

The CPython Stable ABI is a promise that extensions built against a
specific Stable ABI version will be compatible with any newer
interpreter of the same major version.

The Stable ABI does not define a complete binary interface: important
details like the layout of structures in memory or function calling
conventions are determined by the platform and the compiler and its
settings. The Stable ABI promise only applies if these lower-details are
also stable.

For example, an extension built with the CPython 3.10 Stable ABI will be
usable with CPython 3.11, 3.12, etc. It will not necessarily be
compatible with CPython 4.0, nor with CPython 3.10 on a different
platform.

The Stable ABI is not generally forward-compatible: an extension built
and tested with CPython 3.10 will not generally be compatible with
CPython 3.9.

Note

For example, starting in Python 3.10, the Py_tp_doc slot may be set to
NULL, while in older versions, a NULL value will likely crash the
interpreter.

The Stable ABI trades performance for its stability. For example,
extensions built for a specific CPython version will automatically use
faster macros instead of functions in the Stable ABI.

Future Python versions may deprecate some members of the Stable ABI.
Deprecated members will still work, but may suffer from issues like
reduced performance or, in the most extreme cases, memory/resource
leaks.

Limited API

The Stable ABI promise holds for extensions compiled from code that
restricts itself to the Limited API (application programming interface).
The Limited API is a subset of CPython's C API.

Extensions that target the Limited API should define the preprocessor
macro Py_LIMITED_API to either 3 or the current PYTHON_API_VERSION. This
will enable Stable ABI versions of several functions and limit
definitions to the Limited API. (However, note that the macro is not
perfect: due to technical issues or oversights, some non-limited API
might be exposed even with it defined.)

The Limited API is not guaranteed to be stable. In the future, parts of
the Limited API may be deprecated. They may even be removed, as long as
the Stable ABI is kept stable and Python's general backwards
compatibility policy, PEP 387, is followed.

Note

For example, a function declaration might be removed from public header
files but kept in the library. This is currently a possibility for the
future; this PEP does not to propose a concrete process for deprecations
and removals.

The goal for the Limited API is to cover everything needed to interact
with the interpreter. The main reason to not include a public API in the
Limited subset should be that it needs implementation details that
change between CPython versions (like struct memory layouts) – usually
for performance reasons.

The Limited API is not limited to CPython. Other implementations are
encouraged to implement it and help drive its design.

Specification

To make the Stable ABI more useful and robust, the following changes are
proposed.

Stable ABI Manifest

All members of the Stable ABI – functions, typedefs, structs, data,
macros, and constants – will be explicitly listed in a single "manifest"
file, Misc/stable_abi.txt.

For structs, any fields that users of the Stable ABI are allowed to
access will be listed explicitly.

The manifest will also serve as the definitive list of the Limited API.
Members that are not part of the Limited API, but are part of the Stable
ABI (e.g. PyObject.ob_type, which is accessible by the Py_TYPE macro),
will be annotated as such.

For items that are only available on some systems, the manifest will
record the feature macro that determines their presence (such as
MS_WINDOWS or HAVE_FORK). To make the implementation (and usage from
non-C languages) easier, all such macros will be simple names. If a
future item needs a “negative” macro or complex expression (such as a
hypothetical #ifndef MACOSX or #if defined(POSIX) && !defined(LINUX)), a
new feature macro will be derived.

The format of the manifest will be subject to change whenever needed. It
should be consumed only by scripts in the CPython repository. If a
stable list is needed, a script can be added to generate it.

The following will be generated from the ABI manifest:

-   Source for the Windows shared library, PC/python3dll.c.
-   Input for documentation (see below).
-   Test case that checks the runtime availability of symbols (see
    below).

The following will be checked against the Stable ABI manifest as part of
continuous integration:

-   The reference count summary, Doc/data/refcounts.txt, includes all
    function in the Stable ABI (among others).
-   The functions/structs declared and constants/macros defined when
    Python.h is included with Py_LIMITED_API set. (Initially Linux only;
    checks on other systems may be added in the future.)

After the initial implementation, details such as function arguments
will be added and the manifest will be checked for internal consistency
(e.g. all types used in function signatures are part of the API).

Contents of the Stable ABI

The initial Stable ABI manifest will include:

-   The Stable ABI specified in PEP 384.
-   Everything listed in PC/python3dll.c.
-   All structs (struct typedefs) which these functions return or take
    as arguments. (Fields of such structs will not necessarily be
    added.)
-   New type slots, such as Py_am_aiter.
-   The type flags Py_TPFLAGS_DEFAULT, Py_TPFLAGS_BASETYPE,
    Py_TPFLAGS_HAVE_GC, Py_TPFLAGS_METHOD_DESCRIPTOR.
-   The calling conventions METH_* (except deprecated ones).
-   All API needed by macros is the Stable ABI (annotated as not being
    part of the Limited API).

Items that are no longer in CPython when this PEP is accepted will be
removed from the list.

Additional items may be added to the initial manifest according to the
checklist below.

Documenting the Limited API

Notes saying “Part of the Limited API” will be added to Python's
documentation automatically, in a way similar to the notes on functions
that return borrowed references.

A complete list of all members of the Limited API will also be added to
the documentation.

Testing the Stable ABI

An automatically generated test module will be added to ensure that all
symbols included in the Stable ABI are available at compile time.

Changing the Limited API

A checklist for changing the Limited API, including adding new items to
it and removing existing ones, will be added to the Devguide. The
checklist will 1) mention best practices and common pitfalls in Python C
API design and 2) guide the developer around the files that need
changing and scripts that need running when the Limited API is changed.

Below is the initial proposal for the checklist. (After the PEP is
accepted, see the Devguide for the current version.)

Note that the checklist applies to new changes; several items in the
existing Limited API are grandfathered and couldn't be added today.

Design considerations:

-   Make sure the change does not break the Stable ABI of any version of
    Python since 3.5.

-   Make sure no exposed names are private (i.e. begin with an
    underscore).

-   Make sure the new API is well documented.

-   Make sure the types of all parameters and return values of the added
    function(s) and all fields of the added struct(s) are be part of the
    Limited API (or standard C).

-   Make sure the new API and its intended use follows standard C, not
    just features of currently supported platforms. Specifically, follow
    the C dialect specified in PEP 7.

    -   Do not cast a function pointer to void* (a data pointer) or vice
        versa.

-   Make sure the new API follows reference counting conventions.
    (Following them makes the API easier to reason about, and easier use
    in other Python implementations.)

    -   Do not return borrowed references from functions.
    -   Do not steal references to function arguments.

-   Make sure the ownership rules and lifetimes of all applicable struct
    fields, arguments and return values are well defined.

-   Think about ease of use for the user. (In C, ease of use itself is
    not very important; what is useful is reducing boilerplate code
    needed to use the API. Bugs like to hide in boiler plates.)

    -   If a function will be often called with specific value for an
        argument, consider making it default (used when NULL is passed
        in).

-   Think about future extensions: for example, if it's possible that
    future Python versions will need to add a new field to your struct,
    how will that be done?

-   Make as few assumptions as possible about details that might change
    in future CPython versions or differ across C API implementations:

      -   The GIL
      -   Garbage collection
      -   Memory layout of PyObject, lists/tuples and other structures

If following these guidelines would hurt performance, add a fast
function (or macro) to the non-limited API and a stable equivalent to
the Limited API.

If anything is unclear, or you have a good reason to break the
guidelines, consider discussing the change at the capi-sig mailing list.

Procedure:

-   Move the declaration to a header file directly under Include/, into
    a #if !defined(Py_LIMITED_API) || Py_LIMITED_API+0 >= 0x03yy0000
    block (with the yy corresponding to the target CPython version).
-   Make an entry in the Stable ABI manifest, Misc/stable_abi.txt.
-   Regenerate the autogenerated files using make regen-all. (or the
    alternative for non-make platforms)
-   Build Python and run checks using make check-abi. (or the
    alternative for non-make platforms)

Advice for Extenders and Embedders

The following notes will be added to documentation, along with better
information regarding this topic and what guarantees do we offer:

Extension authors should test with all Python versions they support, and
preferably build with the lowest such version.

Compiling with Py_LIMITED_API defined is not a guarantee that your code
conforms to the Limited API or the Stable ABI. Py_LIMITED_API only
covers definitions, but an API also includes other issues, such as
expected semantics.

Examples of issues that Py_LIMITED_API does not guard against are:

-   Calling a function with invalid arguments
-   A function that started accepting NULL values for an argument in
    Python 3.9 will fail if NULL is passed to it under Python 3.8. Only
    testing with 3.8 (or lower versions) will uncover this issue.
-   Some structs include a few fields that are part of the Stable ABI
    and other fields that aren't. Py_LIMITED_API does not filter out
    such “private” fields.
-   Code that uses something that is not documented as part of the
    Stable ABI, but exposed even with Py_LIMITED_API defined, may break
    in the future. Despite the team's best efforts, such issues may
    happen.

Note for Redistributors of Python

The Stable ABI promise relies on stable underlying ABI details, such as
the layout of structures in memory and function calling conventions,
which are affected by the compiler and its settings. For the promise to
hold, these details must not change between CPython 3.x releases on a
particular platform.

Backwards Compatibility

Backwards compatibility is one honking great idea!

This PEP aims at full compatibility with the existing Stable ABI and
Limited API, but defines them terms more explicitly. It might not be
consistent with some interpretations of what the existing Stable
ABI/Limited API is.

Security Implications

None known.

How to Teach This

Technical documentation will be provided in Doc/c-api/stable and linked
from the What's New document. Docs for CPython core developers will be
added to the devguide.

Reference Implementation

See issue 43795.

Ideas for the Future

The following issues are out of scope of this PEP, but show possible
future directions.

Defining a process for deprecations/removals

While this PEP acknowledges that parts of the Limited API might be
deprecated or removed in the future, a process to do this is not in
scope, and is left to a possible future PEP.

C syntax for the ABI manifest

It might be useful to have the ABI manifest be a C header file, or to
generate header files from the manifest. Again, either are options for
the future.

Open Issues

None so far.

References

Copyright

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.