PEP: 741 Title: Python Configuration C API Author: Victor Stinner
<vstinner@python.org> Discussions-To:
https://discuss.python.org/t/pep-741-python-configuration-c-api-second-version/45403
Status: Final Type: Standards Track Created: 18-Jan-2024 Python-Version:
3.14 Post-History: 19-Jan-2024, 08-Feb-2024, Resolution:
https://discuss.python.org/t/pep-741-python-configuration-c-api-second-version/45403/88

Abstract

Add a C API to configure the Python initialization without relying on C
structures and the ability to make ABI-compatible changes in the future.

Complete PEP 587 API by adding PyInitConfig_AddModule() which can be
used to add a built-in extension module; feature previously referred to
as the "inittab".

Add PyConfig_Get() and PyConfig_Set() functions to get and set the
current runtime configuration.

PEP 587 "Python Initialization Configuration" unified all the ways to
configure the Python initialization. This PEP unifies also the
configuration of the Python preinitialization and the Python
initialization in a single API. Moreover, this PEP only provides a
single choice to embed Python, instead of having two "Python" and
"Isolated" choices (PEP 587), to simplify the API further.

The lower level PEP 587 PyConfig API remains available for use cases
with an intentionally higher level of coupling to CPython implementation
details (such as emulating the full functionality of CPython's CLI,
including its configuration mechanisms).

Rationale

Get the runtime configuration

PEP 587 has no API to get the current runtime configuration, only to
configure the Python initialization.

For example, the global configuration variable Py_UnbufferedStdioFlag
was deprecated in Python 3.12 and using PyConfig.buffered_stdio is
recommended instead. It only works to configure Python, there is no
public API to get PyConfig.buffered_stdio.

Users of the limited C API are asking for a public API to get the
current runtime configuration.

Cython needs to get the optimization_level configuration option: issue.

When global configuration variables were deprecated in 2022, Marc-André
Lemburg requested a C API to access these configuration variables at
runtime (not only during Python initialization).

Security fix

To fix CVE-2020-10735, a denial-of-service when converting a very large
string to an integer (in base 10), it was discussed to add a new
PyConfig member to stable branches which affects the ABI.

Gregory P. Smith proposed a different API using text based configuration
file to not be limited by PyConfig members: FR: Allow private runtime
config to enable extending without breaking the PyConfig ABI (August
2022).

In the end, it was decided to not add a new PyConfig member to stable
branches, but only add a new PyConfig.int_max_str_digits member to the
development branch (which became Python 3.12). A dedicated private
global variable (unrelated to PyConfig) is used in stable branches.

Redundancy between PyPreConfig and PyConfig

The Python preinitialization uses the PyPreConfig structure and the
Python initialization uses the PyConfig structure. Both structures have
four duplicated members: dev_mode, parse_argv, isolated and
use_environment.

The redundancy is caused by the fact that the two structures are
separated, whereas some PyConfig members are needed by the
preinitialization.

Embedding Python

Applications embedding Python

Examples:

-   Blender 3D graphics.
-   fontforge font editor.
-   Gimp.
-   LibreOffice.
-   OBS Studio.
-   Tiled.
-   vim text editor.

On Linux, FreeBSD and macOS, applications are usually either statically
linked to a libpython, or load dynamically a libpython . The libpython
shared library is versioned, example: libpython3.12.so for Python 3.12
on Linux.

The vim project can target the stable ABI. Usually, the "system Python"
version is used. It's not currently possible to select which Python
version to use. Users would like the ability to select a newer Python on
demand.

On Linux, another approach to deploy an application embedding Python,
such as GIMP, is to include Python in Flatpack, AppImage or Snap
"container". In this case, the application brings its own copy of Python
version with the container.

Libraries embedding Python

Examples:

-   Apache mod_wsgi (source).
-   nimpy: Nim - Python bridge.
-   PyO3: Rust bindings for the Python interpreter.

Utilities creating standalone applications

-   py2app for macOS.
-   py2exe for Windows.
-   pyinstaller.
-   PyOxidizer: it uses the PEP 587 PyConfig API.

These utilities create standalone applications, they are not linked to
libpython.

Set the runtime configuration

Marc-André Lemburg requested a C API to set the value of some
configuration options at runtime:

-   optimization_level
-   verbose
-   parser_debug
-   inspect
-   write_bytecode

Previously, it was possible to set directly global configuration
variables:

-   Py_OptimizeFlag
-   Py_VerboseFlag
-   Py_DebugFlag
-   Py_InspectFlag
-   Py_DontWriteBytecodeFlag

But these configuration flags were deprecated in Python 3.12 and are
scheduled for removal in Python 3.14.

Specification

Add C API functions and structure to configure the Python
initialization:

-   Create config:
    -   PyInitConfig opaque structure.
    -   PyInitConfig_Create().
    -   PyInitConfig_Free(config).
-   Get options:
    -   PyInitConfig_HasOption(config, name).
    -   PyInitConfig_GetInt(config, name, &value).
    -   PyInitConfig_GetStr(config, name, &value).
    -   PyInitConfig_GetStrList(config, name, &length, &items).
    -   PyInitConfig_FreeStrList().
-   Set options:
    -   PyInitConfig_SetInt(config, name, value).
    -   PyInitConfig_SetStr(config, name, value).
    -   PyInitConfig_SetStrList(config, name, length, items).
    -   PyInitConfig_AddModule(config, name, initfunc)
-   Initialize:
    -   Py_InitializeFromInitConfig(config).
-   Error handling:
    -   PyInitConfig_GetError(config, &err_msg).
    -   PyInitConfig_GetExitcode(config, &exitcode).

Add C API functions to get and set the current runtime configuration:

-   PyConfig_Get(name).
-   PyConfig_GetInt(name, &value).
-   PyConfig_Set(name).
-   PyConfig_Names().

The C API uses null-terminated UTF-8 encoded strings to refer to a
configuration option name.

These C API functions are excluded from the limited C API.

PyInitConfig structure

The PyInitConfig structure is implemented by combining the three
structures of the PyConfig API and has an inittab member as well:

-   PyPreConfig preconfig
-   PyConfig config
-   PyStatus status
-   struct _inittab *inittab for PyInitConfig_AddModule()

The PyStatus status is no longer separated, but part of the unified
PyInitConfig structure, which makes the API easier to use.

Configuration Options

Configuration options are named after PyPreConfig and PyConfig structure
members. See the PyPreConfig documentation and the PyConfig
documentation.

Deprecating and removing configuration options is out of the scope of
the PEP and should be discussed on a case by case basis.

Public configuration options

Following options can be get by PyConfig_Get() and set and
PyConfig_Set().

  ------------------------------------------------------------------------------
  Option                Type             Comment
  --------------------- ---------------- ---------------------------------------
  argv                  list[str]        API: sys.argv.

  base_exec_prefix      str              API: sys.base_exec_prefix.

  base_executable       str              API: sys._base_executable.

  base_prefix           str              API: sys.base_prefix.

  bytes_warning         int              API: sys.flags.bytes_warning.

  exec_prefix           str              API: sys.exec_prefix.

  executable            str              API: sys.executable.

  inspect               bool             API: sys.flags.inspect (int).

  int_max_str_digits    int              API: sys.flags.int_max_str_digits,
                                         sys.get_int_max_str_digits() and
                                         sys.set_int_max_str_digits().

  interactive           bool             API: sys.flags.interactive.

  module_search_paths   list[str]        API: sys.path.

  optimization_level    int              API: sys.flags.optimize.

  parser_debug          bool             API: sys.flags.debug (int).

  platlibdir            str              API: sys.platlibdir.

  prefix                str              API: sys.base_prefix.

  pycache_prefix        str              API: sys.pycache_prefix.

  quiet                 bool             API: sys.flags.quiet (int).

  stdlib_dir            str              API: sys._stdlib_dir.

  use_environment       bool             API: sys.flags.ignore_environment
                                         (int).

  verbose               int              API: sys.flags.verbose.

  warnoptions           list[str]        API: sys.warnoptions.

  write_bytecode        bool             API: sys.flags.dont_write_bytecode
                                         (int) and sys.dont_write_bytecode
                                         (bool).

  xoptions              dict[str, str]   API: sys._xoptions.
  ------------------------------------------------------------------------------

Some option names are different than sys attributes, such as
optimization_level option and sys.flags.optimize attribute.
PyConfig_Set() sets the corresponding sys attribute.

The xoptions is a list of strings in PyInitConfig where each string has
the format key (value is True implicitly) or key=value. In the current
runtime configuration, it becomes a dictionary (key: str →
value: str | True).

Read-only configuration options

Following options can be get by PyConfig_Get(), but cannot be set by
PyConfig_Set().

  ------------------------------------------------------------------------------------
  Option                       Type            Comment
  ---------------------------- --------------- ---------------------------------------
  allocator                    int             

  buffered_stdio               bool            

  check_hash_pycs_mode         str             

  code_debug_ranges            bool            

  coerce_c_locale              bool            

  coerce_c_locale_warn         bool            

  configure_c_stdio            bool            

  configure_locale             bool            

  cpu_count                    int             API: os.cpu_count() (int | None).

  dev_mode                     bool            API: sys.flags.dev_mode.

  dump_refs                    bool            

  dump_refs_file               str             

  faulthandler                 bool            API: faulthandler.is_enabled().

  filesystem_encoding          str             API: sys.getfilesystemencoding().

  filesystem_errors            str             API: sys.getfilesystemencodeerrors().

  hash_seed                    int             

  home                         str             

  import_time                  bool            

  install_signal_handlers      bool            

  isolated                     bool            API: sys.flags.isolated (int).

  legacy_windows_fs_encoding   bool            Windows only.

  legacy_windows_stdio         bool            Windows only.

  malloc_stats                 bool            

  orig_argv                    list[str]       API: sys.orig_argv.

  parse_argv                   bool            

  pathconfig_warnings          bool            

  perf_profiling               bool            API: sys.is_stack_trampoline_active().

  program_name                 str             

  run_command                  str             

  run_filename                 str             

  run_module                   str             

  run_presite                  str             need a debug build.

  safe_path                    bool            

  show_ref_count               bool            

  site_import                  bool            API: sys.flags.no_site (int).

  skip_source_first_line       bool            

  stdio_encoding               str             API: sys.stdin.encoding,
                                               sys.stdout.encoding and
                                               sys.stderr.encoding.

  stdio_errors                 str             API: sys.stdin.errors,
                                               sys.stdout.errors and
                                               sys.stderr.errors.

  tracemalloc                  int             API: tracemalloc.is_tracing() (bool).

  use_frozen_modules           bool            

  use_hash_seed                bool            

  user_site_directory          bool            API: sys.flags.no_user_site (int).

  utf8_mode                    bool            

  warn_default_encoding        bool            

  _pystats                     bool            API: sys._stats_on(), sys._stats_off().
                                               Need a Py_STATS build.
  ------------------------------------------------------------------------------------

Create Config

PyInitConfig structure:

    Opaque structure to configure the Python preinitialization and the
    Python initialization.

PyInitConfig* PyInitConfig_Create(void):

    Create a new initialization configuration using default values of
    the Isolated Configuration.

    It must be freed with PyInitConfig_Free().

    Return NULL on memory allocation failure.

void PyInitConfig_Free(PyInitConfig *config):

    Free memory of an initialization configuration.

Get Options

The configuration option name parameter must be a non-NULL
null-terminated UTF-8 encoded string.

int PyInitConfig_HasOption(PyInitConfig *config, const char *name):

    Test if the configuration has an option called name.

    Return 1 if the option exists, or return 0 otherwise.

int PyInitConfig_GetInt(PyInitConfig *config, const char *name, int64_t *value):

    Get an integer configuration option.

    -   Set *value, and return 0 on success.
    -   Set an error in config and return -1 on error.

int PyInitConfig_GetStr(PyInitConfig *config, const char *name, char **value):

    Get a string configuration option as a null-terminated UTF-8 encoded
    string.

    -   Set *value, and return 0 on success.
    -   Set an error in config and return -1 on error.

    On success, the string must be released with free(value).

int PyInitConfig_GetStrList(PyInitConfig *config, const char *name, size_t *length, char ***items):

    Get a string list configuration option as an array of
    null-terminated UTF-8 encoded strings.

    -   Set *length and *value, and return 0 on success.
    -   Set an error in config and return -1 on error.

    On success, the string list must be released with
    PyInitConfig_FreeStrList(length, items).

void PyInitConfig_FreeStrList(size_t length, char **items):

    Free memory of a string list created by PyInitConfig_GetStrList().

Set Options

The configuration option name parameter must be a non-NULL
null-terminated UTF-8 encoded string.

Some configuration options have side effects on other options. This
logic is only implemented when Py_InitializeFromInitConfig() is called,
not by the "Set" functions below. For example, setting dev_mode to 1
does not set faulthandler to 1.

int PyInitConfig_SetInt(PyInitConfig *config, const char *name, int64_t value):

    Set an integer configuration option.

    -   Return 0 on success.
    -   Set an error in config and return -1 on error.

int PyInitConfig_SetStr(PyInitConfig *config, const char *name, const char *value):

    Set a string configuration option from a null-terminated UTF-8
    encoded string. The string is copied.

    -   Return 0 on success.
    -   Set an error in config and return -1 on error.

int PyInitConfig_SetStrList(PyInitConfig *config, const char *name, size_t length, char * const *items):

    Set a string list configuration option from an array of
    null-terminated UTF-8 encoded strings. The string list is copied.

    -   Return 0 on success.
    -   Set an error in config and return -1 on error.

int PyInitConfig_AddModule(PyInitConfig *config, const char *name, PyObject* (*initfunc)(void)):

    Add a built-in extension module to the table of built-in modules.

    The new module can be imported by the name name, and uses the
    function initfunc as the initialization function called on the first
    attempted import.

    -   Return 0 on success.
    -   Set an error in config and return -1 on error.

    If Python is initialized multiple times, PyInitConfig_AddModule()
    must be called at each Python initialization.

    Similar to the PyImport_AppendInittab() function.

Initialize Python

int Py_InitializeFromInitConfig(PyInitConfig *config):

    Initialize Python from the initialization configuration.

    -   Return 0 on success.
    -   Set an error in config and return -1 on error.
    -   Set an exit code in config and return -1 if Python wants to
        exit.

    See PyInitConfig_GetExitcode() for the exitcode case.

Error Handling

int PyInitConfig_GetError(PyInitConfig* config, const char **err_msg):

    Get the config error message.

    -   Set *err_msg and return 1 if an error is set.
    -   Set *err_msg to NULL and return 0 otherwise.

    An error message is an UTF-8 encoded string.

    If config has an exit code, format the exit code as an error
    message.

    The error message remains valid until another PyInitConfig function
    is called with config. The caller doesn't have to free the error
    message.

int PyInitConfig_GetExitcode(PyInitConfig* config, int *exitcode):

    Get the config exit code.

    -   Set *exitcode and return 1 if Python wants to exit.
    -   Return 0 if config has no exit code set.

    Only the Py_InitializeFromInitConfig() function can set an exit code
    if the parse_argv option is non-zero.

    An exit code can be set when parsing the command line failed (exit
    code 2) or when a command line option asks to display the command
    line help (exit code 0).

Get and Set the Runtime Configuration

The configuration option name parameter must be a non-NULL
null-terminated UTF-8 encoded string.

PyObject* PyConfig_Get(const char *name):

    Get the current runtime value of a configuration option as a Python
    object.

    -   Return a new reference on success.
    -   Set an exception and return NULL on error.

    The object type depends on the option: see Configuration Options
    tables.

    Other options are get from internal PyPreConfig and PyConfig
    structures.

    The caller must hold the GIL. The function cannot be called before
    Python initialization nor after Python finalization.

int PyConfig_GetInt(const char *name, int *value):

    Similar to PyConfig_Get(), but get the value as an integer.

    -   Set *value and return 0 success.
    -   Set an exception and return -1 on error.

PyObject* PyConfig_Names(void):

    Get all configuration option names as a frozenset.

    Set an exception and return NULL on error.

    The caller must hold the GIL.

PyObject* PyConfig_Set(const char *name, PyObject *value):

    Set the current runtime value of a configuration option.

    -   Raise a ValueError if there is no option name.
    -   Raise a ValueError if value is an invalid value.
    -   Raise a ValueError if the option is read-only: cannot be set.
    -   Raise a TypeError if value has not the proper type.

    Read-only configuration options cannot be set.

    The caller must hold the GIL. The function cannot be called before
    Python initialization nor after Python finalization.

Stability

The behavior of options, the default option values, and the Python
behavior can change at each Python version: they are not "stable".

Moreover, configuration options can be added, deprecated and removed
following the usual PEP 387 deprecation process.

Interaction with the PyPreConfig and PyConfig APIs

The lower level PEP 587 PyPreConfig and PyConfig APIs remain available
and fully supported. As noted in the Abstract, they remain the preferred
approach for embedding use cases that are aiming to closely emulate the
behaviour of the full CPython CLI, rather than just making a Python
runtime available as part of a larger application.

The PyPreConfig APIs may be used in combination with the initialization
API in this PEP. In such cases, the read-only vs read/write restrictions
for preconfiguration settings apply to PyInitConfig_SetInt in addition
to PyConfig_Set once the interpreter has been preconfigured
(specifically, only use_environment may be updated, attempting to update
any of the other preconfiguration variables will report an error).

Examples

Initialize Python

Example initializing Python, set configuration options of various types,
return -1 on error:

    int init_python(void)
    {
        PyInitConfig *config = PyInitConfig_Create();
        if (config == NULL) {
            printf("PYTHON INIT ERROR: memory allocation failed\n");
            return -1;
        }

        // Set an integer (dev mode)
        if (PyInitConfig_SetInt(config, "dev_mode", 1) < 0) {
            goto error;
        }

        // Set a list of UTF-8 strings (argv)
        char *argv[] = {"my_program", "-c", "pass"};
        if (PyInitConfig_SetStrList(config, "argv",
                                     Py_ARRAY_LENGTH(argv), argv) < 0) {
            goto error;
        }

        // Set a UTF-8 string (program name)
        if (PyInitConfig_SetStr(config, "program_name", L"my_program") < 0) {
            goto error;
        }

        // Initialize Python with the configuration
        if (Py_InitializeFromInitConfig(config) < 0) {
            goto error;
        }
        PyInitConfig_Free(config);
        return 0;

    error:
        // Display the error message
        const char *err_msg;
        (void)PyInitConfig_GetError(config, &err_msg);
        printf("PYTHON INIT ERROR: %s\n", err_msg);
        PyInitConfig_Free(config);

        return -1;
    }

Increase initialization bytes_warning option

Example increasing the bytes_warning option of an initialization
configuration:

    int config_bytes_warning(PyInitConfig *config)
    {
        int64_t bytes_warning;
        if (PyInitConfig_GetInt(config, "bytes_warning", &bytes_warning)) {
            return -1;
        }
        bytes_warning += 1;
        if (PyInitConfig_SetInt(config, "bytes_warning", bytes_warning)) {
            return -1;
        }
        return 0;
    }

Get the runtime verbose option

Example getting the current runtime value of the configuration option
verbose:

    int get_verbose(void)
    {
        int verbose;
        if (PyConfig_GetInt("verbose", &verbose) < 0) {
            // Silently ignore the error
            PyErr_Clear();
            return -1;
        }
        return verbose;
    }

On error, the function silently ignores the error and returns -1. In
practice, getting the verbose option cannot fail, unless a future Python
version removes the option.

Implementation

-   Issue: [C API] PEP 741: Add PyInitConfig C API to customize the
    Python initialization
-   PR: Add PyConfig_Get() function
-   PR: Add PyInitConfig C API

Backwards Compatibility

Changes are fully backward compatible. Only new APIs are added.

Existing API such as the PyConfig C API (PEP 587) are left unchanged.

Rejected Ideas

Configuration as text

It was proposed to provide the configuration as text to make the API
compatible with the stable ABI and to allow custom options.

Example:

    # integer
    bytes_warning = 2

    # string
    filesystem_encoding = "utf8"   # comment

    # list of strings
    argv = ['python', '-c', 'code']

The API would take the configuration as a string, not as a file. Example
with a hypothetical PyInit_SetConfig() function:

    void stable_abi_init_demo(int set_path)
    {
        PyInit_SetConfig(
            "isolated = 1\n"
            "argv = ['python', '-c', 'code']\n"
            "filesystem_encoding = 'utf-8'\n"
        );
        if (set_path) {
            PyInit_SetConfig("pythonpath = '/my/path'");
        }
    }

The example ignores error handling to make it easier to read.

The problem is that generating such configuration text requires adding
quotes to strings and to escape quotes in strings. Formatting an array
of strings becomes non-trivial.

Providing an API to format a string or an array of strings is not really
worth it, whereas Python can provide directly an API to set a
configuration option where the value is passed directly as a string or
an array of strings. It avoids giving special meaning to some
characters, such as newline characters, which would have to be escaped.

Refer to an option with an integer

Using strings to refer to a configuration option requires comparing
strings which can be slower than comparing integers.

Use integers, similar to type "slots" such as Py_tp_doc, to refer to a
configuration option. The const char *name parameter is replaced with
int option.

Accepting custom options is more likely to cause conflicts when using
integers, since it's harder to maintain "namespaces" (ranges) for
integer options. Using strings, a simple prefix with a colon separator
can be used.

Integers also requires maintaining a list of integer constants and so
make the C API and the Python API larger.

Python 3.13 only has around 62 configuration options, and so performance
is not really a blocker issue. If better performance is needed later, a
hash table can be used to get an option by its name.

If getting a configuration option is used in hot code, the value can be
read once and cached. By the way, most configuration options cannot be
changed at runtime.

Multi-phase initialization (similar to PEP 432)

Eric Snow expressed concerns that this proposal might reinforce with
embedders the idea that initialization is a single monolithic step. He
argued that initialization involves 5 distinct phases and even suggested
that the API should reflect this explicitly. Eric proposed that, at the
very least, the implementation of initialization should reflect the
phases, in part for improved code health. Overall, his explanation has
some similarities with PEP 432 and PEP 587.

Another of Eric's key points relevant to this PEP was that, ideally, the
config passed to Py_InitializeFromConfig() should be complete before
that function is called, whereas currently initialization actually
modifies the config.

While Eric wasn't necessarily suggesting an alternative to PEP 741, any
proposal to add a granular initialization API around phases is
effectively the opposite of what this PEP is trying to accomplish. Such
API is more complicated, it requires adding new public structures and
new public functions. It makes the Python initialization more
complicated, rather than this PEP tries to unify existing APIs and make
them simpler (the opposite). Having multiple structures for similar
purpose can lead to duplicate members, similar issue than duplicated
members between existing PyPreConfig and PyConfig structures.

Locale encoding and wide strings

Accepting strings encoded to the locale encoding and accepting wide
strings (wchar_t*) in the PyInitConfig API was deferred to keep the
PyInitConfig API simple and avoid the complexity of the Python
preinitialization. These features are also mostly needed when emulating
the full CPython CLI behaviour, and hence better served by the lower
level PEP 587 API.

Discussions

-   PEP 741: Python Configuration C API (second version) (February
    2024).
-   PEP 741: Python Configuration C API (January 2024).
-   FR: Allow private runtime config to enable extending without
    breaking the PyConfig ABI (August 2022).

Copyright

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.