PEP: 286 Title: Enhanced Argument Tuples Author: Martin von Löwis
<martin@v.loewis.de> Status: Deferred Type: Standards Track
Content-Type: text/x-rst Created: 03-Mar-2002 Python-Version: 2.3
Post-History:

Abstract

PyArg_ParseTuple is confronted with difficult memory management if an
argument converter creates new memory. To deal with these cases, a
specialized argument type is proposed.

PEP Deferral

Further exploration of the concepts covered in this PEP has been
deferred for lack of a current champion interested in promoting the
goals of the PEP and collecting and incorporating feedback, and with
sufficient available time to do so effectively.

The resolution of this PEP may also be affected by the resolution of PEP
426, which proposes the use of a preprocessing step to generate some
aspects of C API interface code.

Problem description

Today, argument tuples keep references to the function arguments, which
are guaranteed to live as long as the argument tuple exists which is at
least as long as the function call is being executed.

In some cases, parsing an argument will allocate new memory, which is
then to be released by the caller. This has two problems:

1.  In case of failure, the application cannot know what memory to
    release; most callers don't even know that they have the
    responsibility to release that memory. Example for this are the N
    converter (bug #416288[1]) and the es# converter (bug #501716[2]).
2.  Even for successful argument parsing, it is still inconvenient for
    the caller to be responsible for releasing the memory. In some
    cases, this is unnecessarily inefficient. For example, the es
    converter copies the conversion result into memory, even though
    there already is a string object that has the right contents.

Proposed solution

A new type 'argument tuple' is introduced. This type derives from tuple,
adding an __dict__ member (at tp_dictoffset -4). Instances of this type
might get the following attributes:

-   'failobjects', a list of objects which need to be deallocated in
    case of success
-   'okobjects', a list of object which will be released when the
    argument tuple is released

To manage this type, the following functions will be added, and used
appropriately in ceval.c and getargs.c:

-   PyArgTuple_New(int);
-   PyArgTuple_AddFailObject(PyObject*, PyObject*);
-   PyArgTuple_AddFailMemory(PyObject*, void*);
-   PyArgTuple_AddOkObject(PyObject*, PyObject*);
-   PyArgTuple_AddOkMemory(PyObject*, void*);
-   PyArgTuple_ClearFailed(PyObject*);

When argument parsing fails, all fail objects will be released through
Py_DECREF, and all fail memory will be released through PyMem_Free. If
parsing succeeds, the references to the fail objects and fail memory are
dropped, without releasing anything.

When the argument tuple is released, all ok objects and memory will be
released.

If those functions are called with an object of a different type, a
warning is issued and no further action is taken; usage of the affected
converters without using argument tuples is deprecated.

Affected converters

The following converters will add fail memory and fail objects: N, es,
et, es#, et# (unless memory is passed into the converter)

New converters

To simplify Unicode conversion, the e* converters are duplicated as E*
converters (Es, Et, Es#, Et#). The usage of the E* converters is
identical to that of the e* converters, except that the application will
not need to manage the resulting memory. This will be implemented
through registration of Ok objects with the argument tuple. The e*
converters are deprecated.

References

Copyright

This document has been placed in the public domain.

[1] infrequent memory leak in pyexpat
(http://bugs.python.org/issue416288)

[2] "es#" parser marker leaks memory
(http://bugs.python.org/issue501716)