PEP: 367 Title: New Super Version: $Revision$ Last-Modified: $Date$
Author: Calvin Spealman <ironfroggy@gmail.com>, Tim Delaney
<timothy.c.delaney@gmail.com> Status: Superseded Type: Standards Track
Content-Type: text/x-rst Created: 28-Apr-2007 Python-Version: 2.6
Post-History: 28-Apr-2007, 29-Apr-2007, 29-Apr-2007, 14-May-2007

Numbering Note

This PEP has been renumbered to PEP 3135. The text below is the last
version submitted under the old number.

Abstract

This PEP proposes syntactic sugar for use of the super type to
automatically construct instances of the super type binding to the class
that a method was defined in, and the instance (or class object for
classmethods) that the method is currently acting upon.

The premise of the new super usage suggested is as follows:

    super.foo(1, 2)

to replace the old:

    super(Foo, self).foo(1, 2)

and the current __builtin__.super be aliased to __builtin__.__super__
(with __builtin__.super to be removed in Python 3.0).

It is further proposed that assignment to super become a SyntaxError,
similar to the behaviour of None.

Rationale

The current usage of super requires an explicit passing of both the
class and instance it must operate from, requiring a breaking of the DRY
(Don't Repeat Yourself) rule. This hinders any change in class name, and
is often considered a wart by many.

Specification

Within the specification section, some special terminology will be used
to distinguish similar and closely related concepts. "super type" will
refer to the actual builtin type named "super". A "super instance" is
simply an instance of the super type, which is associated with a class
and possibly with an instance of that class.

Because the new super semantics are not backwards compatible with Python
2.5, the new semantics will require a __future__ import:

    from __future__ import new_super

The current __builtin__.super will be aliased to __builtin__.__super__.
This will occur regardless of whether the new super semantics are
active. It is not possible to simply rename __builtin__.super, as that
would affect modules that do not use the new super semantics. In Python
3.0 it is proposed that the name __builtin__.super will be removed.

Replacing the old usage of super, calls to the next class in the MRO
(method resolution order) can be made without explicitly creating a
super instance (although doing so will still be supported via
__super__). Every function will have an implicit local named super. This
name behaves identically to a normal local, including use by inner
functions via a cell, with the following exceptions:

1.  Assigning to the name super will raise a SyntaxError at compile
    time;
2.  Calling a static method or normal function that accesses the name
    super will raise a TypeError at runtime.

Every function that uses the name super, or has an inner function that
uses the name super, will include a preamble that performs the
equivalent of:

    super = __builtin__.__super__(<class>, <instance>)

where <class> is the class that the method was defined in, and
<instance> is the first parameter of the method (normally self for
instance methods, and cls for class methods). For static methods and
normal functions, <class> will be None, resulting in a TypeError being
raised during the preamble.

Note: The relationship between super and __super__ is similar to that
between import and __import__.

Much of this was discussed in the thread of the python-dev list, "Fixing
super anyone?"[1].

Open Issues

Determining the class object to use

The exact mechanism for associating the method with the defining class
is not specified in this PEP, and should be chosen for maximum
performance. For CPython, it is suggested that the class instance be
held in a C-level variable on the function object which is bound to one
of NULL (not part of a class), Py_None (static method) or a class object
(instance or class method).

Should super actually become a keyword?

With this proposal, super would become a keyword to the same extent that
None is a keyword. It is possible that further restricting the super
name may simplify implementation, however some are against the actual
keyword-ization of super. The simplest solution is often the correct
solution and the simplest solution may well not be adding additional
keywords to the language when they are not needed. Still, it may solve
other open issues.

Closed Issues

super used with __call__ attributes

It was considered that it might be a problem that instantiating super
instances the classic way, because calling it would lookup the __call__
attribute and thus try to perform an automatic super lookup to the next
class in the MRO. However, this was found to be false, because calling
an object only looks up the __call__ method directly on the object's
type. The following example shows this in action.

    class A(object):
        def __call__(self):
            return '__call__'
        def __getattribute__(self, attr):
            if attr == '__call__':
                return lambda: '__getattribute__'
    a = A()
    assert a() == '__call__'
    assert a.__call__() == '__getattribute__'

In any case, with the renaming of __builtin__.super to
__builtin__.__super__ this issue goes away entirely.

Reference Implementation

It is impossible to implement the above specification entirely in
Python. This reference implementation has the following differences to
the specification:

1.  New super semantics are implemented using bytecode hacking.
2.  Assignment to super is not a SyntaxError. Also see point #4.
3.  Classes must either use the metaclass autosuper_meta or inherit from
    the base class autosuper to acquire the new super semantics.
4.  super is not an implicit local variable. In particular, for inner
    functions to be able to use the super instance, there must be an
    assignment of the form super = super in the method.

The reference implementation assumes that it is being run on Python
2.5+.

    #!/usr/bin/env python
    #
    # autosuper.py

    from array import array
    import dis
    import new
    import types
    import __builtin__
    __builtin__.__super__ = __builtin__.super
    del __builtin__.super

    # We need these for modifying bytecode
    from opcode import opmap, HAVE_ARGUMENT, EXTENDED_ARG

    LOAD_GLOBAL = opmap['LOAD_GLOBAL']
    LOAD_NAME = opmap['LOAD_NAME']
    LOAD_CONST = opmap['LOAD_CONST']
    LOAD_FAST = opmap['LOAD_FAST']
    LOAD_ATTR = opmap['LOAD_ATTR']
    STORE_FAST = opmap['STORE_FAST']
    LOAD_DEREF = opmap['LOAD_DEREF']
    STORE_DEREF = opmap['STORE_DEREF']
    CALL_FUNCTION = opmap['CALL_FUNCTION']
    STORE_GLOBAL = opmap['STORE_GLOBAL']
    DUP_TOP = opmap['DUP_TOP']
    POP_TOP = opmap['POP_TOP']
    NOP = opmap['NOP']
    JUMP_FORWARD = opmap['JUMP_FORWARD']
    ABSOLUTE_TARGET = dis.hasjabs

    def _oparg(code, opcode_pos):
        return code[opcode_pos+1] + (code[opcode_pos+2] << 8)

    def _bind_autosuper(func, cls):
        co = func.func_code
        name = func.func_name
        newcode = array('B', co.co_code)
        codelen = len(newcode)
        newconsts = list(co.co_consts)
        newvarnames = list(co.co_varnames)

        # Check if the global 'super' keyword is already present
        try:
            sn_pos = list(co.co_names).index('super')
        except ValueError:
            sn_pos = None

        # Check if the varname 'super' keyword is already present
        try:
            sv_pos = newvarnames.index('super')
        except ValueError:
            sv_pos = None

        # Check if the cellvar 'super' keyword is already present
        try:
            sc_pos = list(co.co_cellvars).index('super')
        except ValueError:
            sc_pos = None

        # If 'super' isn't used anywhere in the function, we don't have anything to do
        if sn_pos is None and sv_pos is None and sc_pos is None:
            return func

        c_pos = None
        s_pos = None
        n_pos = None

        # Check if the 'cls_name' and 'super' objects are already in the constants
        for pos, o in enumerate(newconsts):
            if o is cls:
                c_pos = pos

            if o is __super__:
                s_pos = pos

            if o == name:
                n_pos = pos

        # Add in any missing objects to constants and varnames
        if c_pos is None:
            c_pos = len(newconsts)
            newconsts.append(cls)

        if n_pos is None:
            n_pos = len(newconsts)
            newconsts.append(name)

        if s_pos is None:
            s_pos = len(newconsts)
            newconsts.append(__super__)

        if sv_pos is None:
            sv_pos = len(newvarnames)
            newvarnames.append('super')

        # This goes at the start of the function. It is:
        #
        #   super = __super__(cls, self)
        #
        # If 'super' is a cell variable, we store to both the
        # local and cell variables (i.e. STORE_FAST and STORE_DEREF).
        #
        preamble = [
            LOAD_CONST, s_pos & 0xFF, s_pos >> 8,
            LOAD_CONST, c_pos & 0xFF, c_pos >> 8,
            LOAD_FAST, 0, 0,
            CALL_FUNCTION, 2, 0,
        ]

        if sc_pos is None:
            # 'super' is not a cell variable - we can just use the local variable
            preamble += [
                STORE_FAST, sv_pos & 0xFF, sv_pos >> 8,
            ]
        else:
            # If 'super' is a cell variable, we need to handle LOAD_DEREF.
            preamble += [
                DUP_TOP,
                STORE_FAST, sv_pos & 0xFF, sv_pos >> 8,
                STORE_DEREF, sc_pos & 0xFF, sc_pos >> 8,
            ]

        preamble = array('B', preamble)

        # Bytecode for loading the local 'super' variable.
        load_super = array('B', [
            LOAD_FAST, sv_pos & 0xFF, sv_pos >> 8,
        ])

        preamble_len = len(preamble)
        need_preamble = False
        i = 0

        while i < codelen:
            opcode = newcode[i]
            need_load = False
            remove_store = False

            if opcode == EXTENDED_ARG:
                raise TypeError("Cannot use 'super' in function with EXTENDED_ARG opcode")

            # If the opcode is an absolute target it needs to be adjusted
            # to take into account the preamble.
            elif opcode in ABSOLUTE_TARGET:
                oparg = _oparg(newcode, i) + preamble_len
                newcode[i+1] = oparg & 0xFF
                newcode[i+2] = oparg >> 8

            # If LOAD_GLOBAL(super) or LOAD_NAME(super) then we want to change it into
            # LOAD_FAST(super)
            elif (opcode == LOAD_GLOBAL or opcode == LOAD_NAME) and _oparg(newcode, i) == sn_pos:
                need_preamble = need_load = True

            # If LOAD_FAST(super) then we just need to add the preamble
            elif opcode == LOAD_FAST and _oparg(newcode, i) == sv_pos:
                need_preamble = need_load = True

            # If LOAD_DEREF(super) then we change it into LOAD_FAST(super) because
            # it's slightly faster.
            elif opcode == LOAD_DEREF and _oparg(newcode, i) == sc_pos:
                need_preamble = need_load = True

            if need_load:
                newcode[i:i+3] = load_super

            i += 1

            if opcode >= HAVE_ARGUMENT:
                i += 2

        # No changes needed - get out.
        if not need_preamble:
            return func

        # Our preamble will have 3 things on the stack
        co_stacksize = max(3, co.co_stacksize)

        # Conceptually, our preamble is on the `def` line.
        co_lnotab = array('B', co.co_lnotab)

        if co_lnotab:
            co_lnotab[0] += preamble_len

        co_lnotab = co_lnotab.tostring()

        # Our code consists of the preamble and the modified code.
        codestr = (preamble + newcode).tostring()

        codeobj = new.code(co.co_argcount, len(newvarnames), co_stacksize,
                           co.co_flags, codestr, tuple(newconsts), co.co_names,
                           tuple(newvarnames), co.co_filename, co.co_name,
                           co.co_firstlineno, co_lnotab, co.co_freevars,
                           co.co_cellvars)

        func.func_code = codeobj
        func.func_class = cls
        return func

    class autosuper_meta(type):
        def __init__(cls, name, bases, clsdict):
            UnboundMethodType = types.UnboundMethodType

            for v in vars(cls):
                o = getattr(cls, v)
                if isinstance(o, UnboundMethodType):
                    _bind_autosuper(o.im_func, cls)

    class autosuper(object):
        __metaclass__ = autosuper_meta

    if __name__ == '__main__':
        class A(autosuper):
            def f(self):
                return 'A'

        class B(A):
            def f(self):
                return 'B' + super.f()

        class C(A):
            def f(self):
                def inner():
                    return 'C' + super.f()

                # Needed to put 'super' into a cell
                super = super
                return inner()

        class D(B, C):
            def f(self, arg=None):
                var = None
                return 'D' + super.f()

        assert D().f() == 'DBCA'

Disassembly of B.f and C.f reveals the different preambles used when
super is simply a local variable compared to when it is used by an inner
function.

    >>> dis.dis(B.f)

    214           0 LOAD_CONST               4 (<type 'super'>)
                  3 LOAD_CONST               2 (<class '__main__.B'>)
                  6 LOAD_FAST                0 (self)
                  9 CALL_FUNCTION            2
                 12 STORE_FAST               1 (super)

    215          15 LOAD_CONST               1 ('B')
                 18 LOAD_FAST                1 (super)
                 21 LOAD_ATTR                1 (f)
                 24 CALL_FUNCTION            0
                 27 BINARY_ADD
                 28 RETURN_VALUE

    >>> dis.dis(C.f)

    218           0 LOAD_CONST               4 (<type 'super'>)
                  3 LOAD_CONST               2 (<class '__main__.C'>)
                  6 LOAD_FAST                0 (self)
                  9 CALL_FUNCTION            2
                 12 DUP_TOP
                 13 STORE_FAST               1 (super)
                 16 STORE_DEREF              0 (super)

    219          19 LOAD_CLOSURE             0 (super)
                 22 LOAD_CONST               1 (<code object inner at 00C160A0, file "autosuper.py", line 219>)
                 25 MAKE_CLOSURE             0
                 28 STORE_FAST               2 (inner)

    223          31 LOAD_FAST                1 (super)
                 34 STORE_DEREF              0 (super)

    224          37 LOAD_FAST                2 (inner)
                 40 CALL_FUNCTION            0
                 43 RETURN_VALUE

Note that in the final implementation, the preamble would not be part of
the bytecode of the method, but would occur immediately following
unpacking of parameters.

Alternative Proposals

No Changes

Although its always attractive to just keep things how they are, people
have sought a change in the usage of super calling for some time, and
for good reason, all mentioned previously.

-   Decoupling from the class name (which might not even be bound to the
    right class anymore!)
-   Simpler looking, cleaner super calls would be better

Dynamic attribute on super type

The proposal adds a dynamic attribute lookup to the super type, which
will automatically determine the proper class and instance parameters.
Each super attribute lookup identifies these parameters and performs the
super lookup on the instance, as the current super implementation does
with the explicit invocation of a super instance upon a class and
instance.

This proposal relies on sys._getframe(), which is not appropriate for
anything except a prototype implementation.

super(__this_class__, self)

This is nearly an anti-proposal, as it basically relies on the
acceptance of the __this_class__ PEP, which proposes a special name that
would always be bound to the class within which it is used. If that is
accepted, __this_class__ could simply be used instead of the class' name
explicitly, solving the name binding issues[2].

self.__super__.foo(*args)

The __super__ attribute is mentioned in this PEP in several places, and
could be a candidate for the complete solution, actually using it
explicitly instead of any super usage directly. However,
double-underscore names are usually an internal detail, and attempted to
be kept out of everyday code.

super(self, *args) or __super__(self, *args)

This solution only solves the problem of the type indication, does not
handle differently named super methods, and is explicit about the name
of the instance. It is less flexible without being able to enacted on
other method names, in cases where that is needed. One use case this
fails is where a base-class has a factory classmethod and a subclass has
two factory classmethods, both of which needing to properly make super
calls to the one in the base-class.

super.foo(self, *args)

This variation actually eliminates the problems with locating the proper
instance, and if any of the alternatives were pushed into the spotlight,
I would want it to be this one.

super or super()

This proposal leaves no room for different names, signatures, or
application to other classes, or instances. A way to allow some similar
use alongside the normal proposal would be favorable, encouraging good
design of multiple inheritance trees and compatible methods.

super(*p, **kw)

There has been the proposal that directly calling super(*p, **kw) would
be equivalent to calling the method on the super object with the same
name as the method currently being executed i.e. the following two
methods would be equivalent:

    def f(self, *p, **kw):
        super.f(*p, **kw)

    def f(self, *p, **kw):
        super(*p, **kw)

There is strong sentiment for and against this, but implementation and
style concerns are obvious. Guido has suggested that this should be
excluded from this PEP on the principle of KISS (Keep It Simple Stupid).

History

29-Apr-2007 - Changed title from "Super As A Keyword" to "New Super"

    -   Updated much of the language and added a terminology section for
        clarification in confusing places.
    -   Added reference implementation and history sections.

06-May-2007 - Updated by Tim Delaney to reflect discussions on the python-3000

    and python-dev mailing lists.

References

Copyright

This document has been placed in the public domain.



  Local Variables: mode: indented-text indent-tabs-mode: nil
  sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

[1] Fixing super anyone?
(https://mail.python.org/pipermail/python-3000/2007-April/006667.html)

[2] PEP 3130: Access to Module/Class/Function Currently Being Defined
(this)
(https://mail.python.org/pipermail/python-ideas/2007-April/000542.html)