PEP: 576 Title: Rationalize Built-in function classes Author: Mark
Shannon <mark@hotpy.org> BDFL-Delegate: Petr Viktorin Status: Withdrawn
Type: Standards Track Content-Type: text/x-rst Created: 10-May-2018
Python-Version: 3.8 Post-History: 17-May-2018, 23-Jun-2018, 08-Jul-2018,
29-Mar-2019

Abstract

Expose the "FastcallKeywords" convention used internally by CPython to
third-party code, and make the inspect module use duck-typing. In
combination this will allow third-party C extensions and tools like
Cython to create objects that use the same calling conventions as
built-in and Python functions, thus gaining performance parity with
built-in functions like len or print.

A small improvement in the performance of existing code is expected.

Motivation

Currently third-party module authors face a dilemma when implementing
functions in C. Either they can use one of the pre-existing built-in
function or method classes or implement their own custom class in C. The
first choice causes them to lose the ability to access the internals of
the callable object. The second choice is an additional maintenance
burden and, more importantly, has a significant negative impact on
performance.

This PEP aims to allow authors of third-party C modules, and tools like
to Cython, to utilize the faster calling convention used internally by
CPython for built-in functions and methods, and to do so without a loss
of capabilities relative to a function implemented in Python.

Introspection

The inspect module will fully support duck-typing when introspecting
callables.

The inspect.Signature.from_callable() function computes the signature of
a callable. If an object has a __signature__ property, then
inspect.Signature.from_callable() simply returns that. To further
support duck-typing, if a callable has a __text_signature__ then the
__signature__ will be created from that.

This means that 3rd party builtin-functions can implement
__text_signature__ if sufficient, and the more expensive __signature__
if necessary.

Efficient calls to third-party callables

Currently the majority of calls are dispatched to functions and
method_descriptors in custom code, using the "FastcallKeywords" internal
calling convention. This PEP proposes that this calling convention is
implemented via a C function pointer. Third-party callables which
implement this binary interface will have the potential to be called as
fast as a built-in function.

Continued prohibition of callable classes as base classes

Currently any attempt to use function, method or method_descriptor as a
base class for a new class will fail with a TypeError. This behaviour is
desirable as it prevents errors when a subclass overrides the __call__
method. If callables could be sub-classed then any call to a function or
a method_descriptor would need an additional check that the __call__
method had not been overridden. By exposing an additional call
mechanism, the potential for errors becomes greater. As a consequence,
any third-party class implementing the addition call interface will not
be usable as a base class.

New classes and changes to existing classes

Python visible changes

1.  A new built-in class, builtin_function, will be added.
2.  types.BuiltinFunctionType will refer to builtin_function not
    builtin_function_or_method.
3.  Instances of the builtin_function class will retain the __module__
    property of builtin_function_or_method and gain the func_module and
    func_globals properties. The func_module allows access to the module
    to which the function belongs. Note that this is different from the
    __module__ property which merely returns the name of the module. The
    func_globals property is equivalent to func_module.__dict__ and is
    provided to mimic the Python function property of the same name.
4.  When binding a method_descriptor instance to an instance of its
    owning class, a bound_method will be created instead of a
    builtin_function_or_method. This means that the method_descriptors
    now mimic the behaviour of Python functions more closely. In other
    words, [].append becomes a bound_method instead of a
    builtin_function_or_method.

C API changes

1.  A new function
    PyBuiltinFunction_New(PyMethodDef *ml, PyObject *module) is added to
    create built-in functions.
2.  PyCFunction_NewEx() and PyCFunction_New() are deprecated and will
    return a PyBuiltinFunction if able, otherwise a
    builtin_function_or_method.

Retaining backwards compatibility in the C API and ABI

The proposed changes are fully backwards and forwards compatible at both
the API and ABI level.

Internal C changes

Two new flags will be allowed for the typeobject.tp_flags field. These
are Py_TPFLAGS_EXTENDED_CALL and Py_TPFLAGS_FUNCTION_DESCRIPTOR

Py_TPFLAGS_EXTENDED_CALL

For any built-in class that sets Py_TPFLAGS_EXTENDED_CALL The C struct
corresponding to this built-in class must begin with the struct
PyExtendedCallable which is defined as follows:

    typedef PyObject *(*extended_call_ptr)(PyObject *callable, PyObject** args,
                       int positional_argcount, PyTupleObject* kwnames);

    typedef struct {
        PyObject_HEAD
        extended_call_ptr ext_call;
    } PyExtendedCallable;

Any class that sets the Py_TPFLAGS_EXTENDED_CALL cannot be used as a
base class and a TypeError will be raised if any Python code tries to
use it a base class.

Py_TPFLAGS_FUNCTION_DESCRIPTOR

If this flag is set for a built-in class F, then instances of that class
are expected to behave the same as a Python function when used as a
class attribute. Specifically, this mean that the value of c.m where C.m
is an instanceof the built-in class F (and c is an instance of C) must
be a bound-method binding C.m and c. Without this flag, it would be
impossible for custom callables to behave like Python functions and be
efficient as Python or built-in functions.

Changes to existing C structs

The function, method_descriptor and method classes will have their
corresponding structs changed to start with the PyExtendedCallable
struct.

Third-party built-in classes using the new extended call interface

To enable call performance on a par with Python functions and built-in
functions, third-party callables should set the Py_TPFLAGS_EXTENDED_CALL
bit of tp_flags and ensure that the corresponding C struct starts with
the PyExtendedCallable. Any built-in class that has the
Py_TPFLAGS_EXTENDED_CALL bit set must also implement the tp_call
function and make sure its behaviour is consistent with the ext_call
function.

Performance implications of these changes

Adding a function pointer to each callable, rather than each class of
callable, enables the choice of dispatching function (the code to
shuffle arguments about and do error checking) to be made when the
callable object is created rather than when it is called. This should
reduce the number of instructions executed between the call-site in the
interpreter and the execution of the callee.

Alternative Suggestions

PEP 580 is an alternative approach to solving the same problem as this
PEP.

Reference implementation

A draft implementation can be found at
https://github.com/markshannon/cpython/tree/pep-576-minimal

Copyright

This document has been placed in the public domain.