PEP: 689 Title: Unstable C API tier Author: Petr Viktorin
<encukou@gmail.com> Discussions-To:
https://discuss.python.org/t/pep-689-unstable-c-api-tier/20452 Status:
Final Type: Standards Track Content-Type: text/x-rst Requires: 523
Created: 22-Apr-2022 Python-Version: 3.12 Post-History: 27-Apr-2022,
25-Aug-2022, 27-Oct-2022, Resolution:
https://discuss.python.org/t/pep-689-unstable-c-api-tier/20452/13

devguide:c-api

User-facing documentation is at py3.12:unstable-c-api.

Abstract

Some functions and types of the C-API are designated unstable, meaning
that they will not change in patch (bugfix/security) releases, but may
change between minor releases (e.g. between 3.11 and 3.12) without
deprecation warnings.

Any C API with a leading underscore is designated internal, meaning that
it may change or disappear without any notice.

Motivation & Rationale

Unstable C API tier

The Python C-API is currently divided into three stability tiers:

-   Limited API, with high compatibility expectations
-   Public API, which follows the backwards compatibility policy
    <387>, and requires deprecation warnings before changes
-   Internal (private) API, which can change at any time.

Tools requiring access to CPython internals (e.g. advanced debuggers and
JIT compilers) are often built for minor series releases of CPython, and
assume that the C-API internals used do not change in patch releases. To
support these tools, we need a tier between the Public and Private
C-API, with guarantees on stability throughout the minor-series release:
the proposed Unstable tier.

Some functions, like PyCode_New(), are documented as unstable (“Calling
[it] directly can bind you to a precise Python version”), and also often
change in practice. The unstable tier should make their status obvious
even to people who don't read the docs carefully enough, making them
hard to use accidentally.

Reserving leading underscores for Private API

Currently, CPython developers don't agree on the exact meaning of a
leading underscore in API names. It is used to mean two different
things:

-   API that may change between minor releases, as in the Unstable tier
    proposed here (e.g. functions introduced in PEP 523).
-   API that is private and should not be used outside of CPython at all
    (e.g. because it may change without notice, or it relies on
    undocumented assumptions that non-CPython code cannot guarantee).

The unclear meaning makes the underscore less useful than it could be.
If it only marked private API, CPython developers could change
underscored functions, or remove unused ones, without researching how
they're documented or used outside CPython.

With the introduction of a dedicated unstable tier, we can clarify the
meaning of the leading underscore. It should mark private API only.

Not breaking code unnecessarily

This PEP specifies that API in the unstable tier should have a special
name prefix. This means functions (macros, etc.) will need to be
renamed. After a rename, the old name should continue to be available
until an incompatible change is made (i.e. until call sites need to be
updated anyway). In other words, just changing the tier of a function
shouldn't break users' code.

Specification

The C API is divided by stability expectations into three “sections”
(internal, public, and limited). We'll now call these stability tiers,
or tiers for short.

An Unstable tier will be added.

APIs (functions, types, etc.) in this tier will named with the
PyUnstable_ prefix, with no leading underscore.

They will be declared in headers used for public API (Include/*.h,
rather than in a subdirectory like Include/unstable/).

Several rules for dealing with the unstable tier will be introduced:

-   Unstable API should have no backwards-incompatible changes across
    patch releases, but may change or be removed in minor releases
    (3.x.0, including Alpha and Beta releases of 3.x.0). Such changes
    must be documented and mentioned in the What's New document.

-   Backwards-incompatible changes to these APIs should be made so that
    code that uses them will need to be updated to compile with the new
    version (e.g. arguments should be added/removed, or a function
    should be renamed, but the semantic meaning of an argument should
    not change).

-   Unstable API should be documented and tested.

-   To move an API from the public tier to the unstable tier, it should
    be exposed under the new PyUnstable_* name.

    The old name should be deprecated (e.g. with Py_DEPRECATED), but
    continue to be available until an incompatible change is made to the
    API. Per Python's backwards compatibility policy (PEP 387), this
    deprecation needs to last at least two releases (without an SC
    exceptions). But it can also last indefinitely -- for example, if
    PEP 590's “provisional” <590#finalizing-the-api>
    _PyObject_Vectorcall was added today, it would be initially named
    PyUnstable_Object_Vectorcall and there would be no plan to remove
    this name.

    In the following cases, an incompatible change (and thus removing
    the deprecated name) is allowed without an SC exception, as if the
    function was already part of the Unstable tier:

    -   Any API introduced before Python 3.12 that is documented to be
        less stable than default.
    -   Any API introduced before Python 3.12 that was named with a
        leading underscore.

    For examples, see the initial unstable API specified in this PEP.

-   To move an internal API to the unstable tier, it should be exposed
    under the new PyUnstable_* name.

    If the old name is documented, or widely used externally, it should
    continue to be available until an incompatible change is made (and
    call sites need to be updated). It should start raising deprecation
    warnings (e.g. using Py_DEPRECATED).

-   To move an API from the unstable tier to the public tier, it should
    be exposed without the PyUnstable_* prefix.

    The old name should remain available until the API is deprecated or
    removed.

-   Adding new unstable API for existing features is allowed even after
    Beta feature freeze, up until the first Release Candidate. Consensus
    on Core Development Discourse or is needed in the Beta period.

These rules will be documented in the devguide, and user documentation
will be updated accordingly.

Reference docs for C API named PyUnstable_* will automatically show
notes with links to the unstable tier documentation.

Leading underscore

C API named with a leading underscore, as well as API only available
with Py_BUILD_CORE, will be considered internal. This means:

-   It may change or be removed without notice in minor releases (3.x.0,
    including Alpha and Beta releases of 3.x.0). API changes in patch
    releases or Release Candidates should only be done if absolutely
    necessary.

-   It should be documented in source comments or Devguide only, not in
    the public documentation.

-   API introduced before Python 3.12 that is documented or widely used
    externally should be moved to the Unstable tier as explained above.

    This might happen long after this PEP is accepted. Consequently, for
    a few years core devs should do some research before changing
    underscored API, especially if it doesn't need Py_BUILD_CORE.

Users of the C API are encouraged to search their codebase for _Py and
_PY identifier prefixes, and treat any hits as issues to be eventually
fixed -- either by switching to an existing alternative, or by opening a
CPython issue to request exposing public API for their use case, and
eventually switching to that.

Initial unstable API

The following API will be moved to the Unstable tier in the initial
implementation as proof of the concept.

Code object constructors:

-   PyUnstable_Code_New() (renamed from PyCode_New)
-   PyUnstable_Code_NewWithPosOnlyArgs() (renamed from
    PyCode_NewWithPosOnlyArgs)

Code extra information (PEP 523):

-   PyUnstable_Eval_RequestCodeExtraIndex() (renamed from
    _PyEval_RequestCodeExtraIndex)
-   PyUnstable_Code_GetExtra() (renamed from _PyCode_GetExtra)
-   PyUnstable_Code_SetExtra() (renamed from _PyCode_SetExtra)

More are expected in Python 3.12, without the need for another PEP.

Backwards Compatibility

The C API backwards compatibility expectations will be made clearer.

All renamed API will be available under old names for as long as
feasible.

How to Teach This

The changes affect advanced C programmers, who should consult the
updated reference documentation, devguide and/or What's New document.

Reference Implementation

https://github.com/python/cpython/compare/main...encukou:unstable-tier

Rejected Ideas

No special prefix

In the initial version of this PEP, unstable API didn't have the
PyUnstable prefix. Instead, defining Py_USING_UNSTABLE_API made the API
available in a given source file, signifying acknowledgement that the
file as a whole will potentially need to be revisited for each Python
release.

However, it was decided that unstable-ness needs to be exposed in the
individual names.

Underscore prefix

It would be possible to mark both private and unstable API with leading
underscores. However, that would dilute the meaning of _Py prefix.
Reserving the prefix for internal API only makes it trivial to search
for.

New header directory

Other API tiers have dedicated directories for headers
(Include/cpython/, Include/internal/).

Since the unstable tier uses a very obvious naming convention and the
names are always available, a directory like Include/unstable/ is
unnecessary.

Python API

It might be good to add a similar tier in the Python (not C) API, e.g.
for types.CodeType. However, the mechanism for that would need to be
different. This is outside the scope of the PEP.

Copyright

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.