PEP: 490 Title: Chain exceptions at C level Version: $Revision$
Last-Modified: $Date$ Author: Victor Stinner <vstinner@python.org>
Status: Rejected Type: Standards Track Content-Type: text/x-rst Created:
25-Mar-2015 Python-Version: 3.6

Abstract

Chain exceptions at C level, as already done at Python level.

Rationale

Python 3 introduced a new killer feature: exceptions are chained by
default, PEP 3134.

Example:

    try:
        raise TypeError("err1")
    except TypeError:
        raise ValueError("err2")

Output:

    Traceback (most recent call last):
      File "test.py", line 2, in <module>
        raise TypeError("err1")
    TypeError: err1

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "test.py", line 4, in <module>
        raise ValueError("err2")
    ValueError: err2

Exceptions are chained by default in Python code, but not in extensions
written in C.

A new private _PyErr_ChainExceptions() function was introduced in Python
3.4.3 and 3.5 to chain exceptions. Currently, it must be called
explicitly to chain exceptions and its usage is not trivial.

Example of _PyErr_ChainExceptions() usage from the zipimport module to
chain the previous OSError to a new ZipImportError exception:

    PyObject *exc, *val, *tb;
    PyErr_Fetch(&exc, &val, &tb);
    PyErr_Format(ZipImportError, "can't open Zip file: %R", archive);
    _PyErr_ChainExceptions(exc, val, tb);

This PEP proposes to also chain exceptions automatically at C level to
stay consistent and give more information on failures to help debugging.
The previous example becomes simply:

    PyErr_Format(ZipImportError, "can't open Zip file: %R", archive);

Proposal

Modify PyErr*() functions to chain exceptions

Modify C functions raising exceptions of the Python C API to
automatically chain exceptions: modify PyErr_SetString(),
PyErr_Format(), PyErr_SetNone(), etc.

Modify functions to not chain exceptions

Keeping the previous exception is not always interesting when the new
exception contains information of the previous exception or even more
information, especially when the two exceptions have the same type.

Example of an useless exception chain with int(str):

    TypeError: a bytes-like object is required, not 'type'

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "<stdin>", line 1, in <module>
    TypeError: int() argument must be a string, a bytes-like object or a number, not 'type'

The new TypeError exception contains more information than the previous
exception. The previous exception should be hidden.

The PyErr_Clear() function can be called to clear the current exception
before raising a new exception, to not chain the current exception with
a new exception.

Modify functions to chain exceptions

Some functions save and then restore the current exception. If a new
exception is raised, the exception is currently displayed into
sys.stderr or ignored depending on the function. Some of these functions
should be modified to chain exceptions instead.

Examples of function ignoring the new exception(s):

-   ptrace_enter_call(): ignore exception
-   subprocess_fork_exec(): ignore exception raised by enable_gc()
-   t_bootstrap() of the _thread module: ignore exception raised by
    trying to display the bootstrap function to sys.stderr
-   PyDict_GetItem(), _PyDict_GetItem_KnownHash(): ignore exception
    raised by looking for a key in the dictionary
-   _PyErr_TrySetFromCause(): ignore exception
-   PyFrame_LocalsToFast(): ignore exception raised by dict_to_map()
-   _PyObject_Dump(): ignore exception. _PyObject_Dump() is used to
    debug, to inspect a running process, it should not modify the Python
    state.
-   Py_ReprLeave(): ignore exception "because there is no way to report
    them"
-   type_dealloc(): ignore exception raised by remove_all_subclasses()
-   PyObject_ClearWeakRefs(): ignore exception?
-   call_exc_trace(), call_trace_protected(): ignore exception
-   remove_importlib_frames(): ignore exception
-   do_mktuple(), helper used by Py_BuildValue() for example: ignore
    exception?
-   flush_io(): ignore exception
-   sys_write(), sys_format(): ignore exception
-   _PyTraceback_Add(): ignore exception
-   PyTraceBack_Print(): ignore exception

Examples of function displaying the new exception to sys.stderr:

-   atexit_callfuncs(): display exceptions with PyErr_Display() and
    return the latest exception, the function calls multiple callbacks
    and only returns the latest exception
-   sock_dealloc(): log the ResourceWarning exception with
    PyErr_WriteUnraisable()
-   slot_tp_del(): display exception with PyErr_WriteUnraisable()
-   _PyGen_Finalize(): display gen_close() exception with
    PyErr_WriteUnraisable()
-   slot_tp_finalize(): display exception raised by the __del__() method
    with PyErr_WriteUnraisable()
-   PyErr_GivenExceptionMatches(): display exception raised by
    PyType_IsSubtype() with PyErr_WriteUnraisable()

Backward compatibility

A side effect of chaining exceptions is that exceptions store traceback
objects which store frame objects which store local variables. Local
variables are kept alive by exceptions. A common issue is a reference
cycle between local variables and exceptions: an exception is stored in
a local variable and the frame indirectly stored in the exception. The
cycle only impacts applications storing exceptions.

The reference cycle can now be fixed with the new
traceback.TracebackException object introduced in Python 3.5. It stores
information required to format a full textual traceback without storing
local variables.

The asyncio is impacted by the reference cycle issue. This module is
also maintained outside Python standard library to release a version for
Python 3.3. traceback.TracebackException will maybe be backported in a
private asyncio module to fix reference cycle issues.

Alternatives

No change

A new private _PyErr_ChainExceptions() function is enough to chain
manually exceptions.

Exceptions will only be chained explicitly where it makes sense.

New helpers to chain exceptions

Functions like PyErr_SetString() don't chain automatically exceptions.
To make the usage of _PyErr_ChainExceptions() easier, new private
functions are added:

-   _PyErr_SetStringChain(exc_type, message)
-   _PyErr_FormatChain(exc_type, format, ...)
-   _PyErr_SetNoneChain(exc_type)
-   _PyErr_SetObjectChain(exc_type, exc_value)

Helper functions to raise specific exceptions like
_PyErr_SetKeyError(key) or PyErr_SetImportError(message, name, path)
don't chain exceptions. The generic
_PyErr_ChainExceptions(exc_type, exc_value, exc_tb) should be used to
chain exceptions with these helper functions.

Appendix

PEPs

-   PEP 3134 -- Exception Chaining and Embedded Tracebacks (Python 3.0):
    new __context__ and __cause__ attributes for exceptions
-   PEP 415 -- Implement context suppression with exception attributes
    (Python 3.3): raise exc from None
-   PEP 409 -- Suppressing exception context (superseded by the PEP 415)

Python C API

The header file Include/pyerror.h declares functions related to
exceptions.

Functions raising exceptions:

-   PyErr_SetNone(exc_type)
-   PyErr_SetObject(exc_type, exc_value)
-   PyErr_SetString(exc_type, message)
-   PyErr_Format(exc, format, ...)

Helpers to raise specific exceptions:

-   PyErr_BadArgument()
-   PyErr_BadInternalCall()
-   PyErr_NoMemory()
-   PyErr_SetFromErrno(exc)
-   PyErr_SetFromWindowsErr(err)
-   PyErr_SetImportError(message, name, path)
-   _PyErr_SetKeyError(key)
-   _PyErr_TrySetFromCause(prefix_format, ...)

Manage the current exception:

-   PyErr_Clear(): clear the current exception, like except: pass
-   PyErr_Fetch(exc_type, exc_value, exc_tb)
-   PyErr_Restore(exc_type, exc_value, exc_tb)
-   PyErr_GetExcInfo(exc_type, exc_value, exc_tb)
-   PyErr_SetExcInfo(exc_type, exc_value, exc_tb)

Others function to handle exceptions:

-   PyErr_ExceptionMatches(exc): check to implement except exc:  ...
-   PyErr_GivenExceptionMatches(exc1, exc2)
-   PyErr_NormalizeException(exc_type, exc_value, exc_tb)
-   _PyErr_ChainExceptions(exc_type, exc_value, exc_tb)

Python Issues

Chain exceptions:

-   Issue #23763: Chain exceptions in C
-   Issue #23696: zipimport: chain ImportError to OSError
-   Issue #21715: Chaining exceptions at C level: added
    _PyErr_ChainExceptions()
-   Issue #18488: sqlite: finalize() method of user function may be
    called with an exception set if a call to step() method failed
-   Issue #23781: Add private _PyErr_ReplaceException() in 2.7
-   Issue #23782: Leak in _PyTraceback_Add

Changes preventing to loose exceptions:

-   Issue #23571: Raise SystemError if a function returns a result with
    an exception set
-   Issue #18408: Fixes crashes found by pyfailmalloc

Rejection

The PEP was rejected on 2017-09-12 by Victor Stinner. It was decided in
the python-dev discussion to not chain C exceptions by default, but
instead chain them explicitly only where it makes sense.

Copyright

This document has been placed in the public domain.



  Local Variables: mode: indented-text indent-tabs-mode: nil
  sentence-end-double-space: t fill-column: 70 coding: utf-8 End: