PEP: 369 Title: Post import hooks Version: $Revision$ Last-Modified:
$Date$ Author: Christian Heimes <christian@python.org> Status: Withdrawn
Type: Standards Track Content-Type: text/x-rst Created: 02-Jan-2008
Python-Version: 2.6, 3.0 Post-History: 02-Dec-2012

Withdrawal Notice

This PEP has been withdrawn by its author, as much of the detailed
design is no longer valid following the migration to importlib in Python
3.3.

Abstract

This PEP proposes enhancements for the import machinery to add post
import hooks. It is intended primarily to support the wider use of
abstract base classes that is expected in Python 3.0.

The PEP originally started as a combined PEP for lazy imports and post
import hooks. After some discussion on the python-dev mailing list the
PEP was parted in two separate PEPs.[1]

Rationale

Python has no API to hook into the import machinery and execute code
after a module is successfully loaded. The import hooks of PEP 302 are
about finding modules and loading modules but they were not designed to
as post import hooks.

Use cases

A use case for a post import hook is mentioned in Alyssa (Nick)
Coghlan's initial posting[2]. about callbacks on module import. It was
found during the development of Python 3.0 and its ABCs. We wanted to
register classes like decimal.Decimal with an ABC but the module should
not be imported on every interpreter startup. Alyssa came up with this
example:

    @imp.when_imported('decimal')
    def register(decimal):
        Inexact.register(decimal.Decimal)

The function register is registered as callback for the module named
'decimal'. When decimal is imported the function is called with the
module object as argument.

While this particular example isn't necessary in practice, (as
decimal.Decimal will inherit from the appropriate abstract Number base
class in 2.6 and 3.0), it still illustrates the principle.

Existing implementations

PJE's peak.util.imports[3] implements post load hooks. My implementation
shares a lot with his and it's partly based on his ideas.

Post import hook implementation

Post import hooks are called after a module has been loaded. The hooks
are callable which take one argument, the module instance. They are
registered by the dotted name of the module, e.g. 'os' or 'os.path'.

The callable are stored in the dict sys.post_import_hooks which is a
mapping from names (as string) to a list of callables or None.

States

No hook was registered

sys.post_import_hooks contains no entry for the module

A hook is registered and the module is not loaded yet

The import hook registry contains an entry sys.post_import_hooks["name"]
= [hook1]

A module is successfully loaded

The import machinery checks if sys.post_import_hooks contains post
import hooks for the newly loaded module. If hooks are found then the
hooks are called in the order they were registered with the module
instance as first argument. The processing of the hooks is stopped when
a method raises an exception. At the end the entry for the module name
set to None, even when an error has occurred.

Additionally the new __notified__ slot of the module object is set to
True in order to prevent infinity recursions when the notification
method is called inside a hook. For object which don't subclass from
PyModule a new attribute is added instead.

A module can't be loaded

The import hooks are neither called nor removed from the registry. It
may be possible to load the module later.

A hook is registered but the module is already loaded

The hook is fired immediately.

Invariants

The import hook system guarantees certain invariants. XXX

Sample Python implementation

A Python implementation may look like:

    def notify(name):
        try:
            module = sys.modules[name]
        except KeyError:
            raise ImportError("Module %s has not been imported" % (name,))
        if module.__notified__:
            return
        try:
            module.__notified__ = True
            if '.' in name:
                notify(name[:name.rfind('.')])
            for callback in post_import_hooks[name]:
               callback(module)
        finally:
            post_import_hooks[name] = None

    XXX

C API

New C API functions

PyObject* PyImport_GetPostImportHooks(void)

    Returns the dict sys.post_import_hooks or NULL

PyObject* PyImport_NotifyLoadedByModule(PyObject *module)

    Notify the post import system that a module was requested. Returns
    the a borrowed reference to the same module object or NULL if an
    error has occurred. The function calls only the hooks for the module
    itself and not its parents. The function must be called with the
    import lock acquired.

PyObject* PyImport_NotifyLoadedByName(const char *name)

    PyImport_NotifyLoadedByName("a.b.c") calls
    PyImport_NotifyLoadedByModule() for a, a.b and a.b.c in that
    particular order. The modules are retrieved from sys.modules. If a
    module can't be retrieved, an exception is raised otherwise the a
    borrowed reference to modname is returned. The hook calls always
    start with the prime parent module. The caller of
    PyImport_NotifyLoadedByName() must hold the import lock!

PyObject* PyImport_RegisterPostImportHook(PyObject *callable, PyObject *mod_name)

    Register a new hook callable for the module mod_name

int PyModule_GetNotified(PyObject *module)

    Returns the status of the __notified__ slot / attribute.

int PyModule_SetNotified(PyObject *module, int status)

    Set the status of the __notified__ slot / attribute.

The PyImport_NotifyLoadedByModule() method is called inside
import_submodule(). The import system makes sure that the import lock is
acquired and the hooks for the parent modules are already called.

Python API

The import hook registry and two new API methods are exposed through the
sys and imp module.

sys.post_import_hooks

    The dict contains the post import hooks:

        {"name" : [hook1, hook2], ...}

imp.register_post_import_hook(hook: "callable", name: str)

    Register a new hook hook for the module name

imp.notify_module_loaded(module: "module instance") -> module

    Notify the system that a module has been loaded. The method is
    provided for compatibility with existing lazy / deferred import
    extensions.

module.__notified__

    A slot of a module instance. XXX

The when_imported function decorator is also in the imp module, which is
equivalent to:

    def when_imported(name):
        def register(hook):
            register_post_import_hook(hook, name)
        return register

imp.when_imported(name) -> decorator function

    for @when_imported(name) def hook(module): pass

Open issues

The when_imported decorator hasn't been written.

The code contains several XXX comments. They are mostly about error
handling in edge cases.

Backwards Compatibility

The new features and API don't conflict with old import system of Python
and don't cause any backward compatibility issues for most software.
However systems like PEAK and Zope which implement their own lazy import
magic need to follow some rules.

The post import hooks carefully designed to cooperate with existing
deferred and lazy import systems. It's the suggestion of the PEP author
to replace own on-load-hooks with the new hook API. The alternative lazy
or deferred imports will still work but the implementations must call
the imp.notify_module_loaded function.

Reference Implementation

A reference implementation is already written and is available in the
py3k-importhook branch.[4] It still requires some cleanups,
documentation updates and additional unit tests.

Acknowledgments

Alyssa Coghlan, for proof reading and the initial discussion Phillip J.
Eby, for his implementation in PEAK and help with my own implementation

Copyright

This document has been placed in the public domain.

References



  Local Variables: mode: indented-text indent-tabs-mode: nil
  sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

[1] PEP: Lazy module imports and post import hook
http://permalink.gmane.org/gmane.comp.python.devel/90949

[2] Interest in PEP for callbacks on module import
http://permalink.gmane.org/gmane.comp.python.python-3000.devel/11126

[3] peak.utils.imports
http://svn.eby-sarna.com/Importing/peak/util/imports.py?view=markup

[4] py3k-importhook branch
http://svn.python.org/view/python/branches/py3k-importhook/