PEP: 3124 Title: Overloading, Generic Functions, Interfaces, and
Adaptation Version: $Revision$ Last-Modified: $Date$ Author: Phillip J.
Eby <pje@telecommunity.com> Discussions-To: python-3000@python.org
Status: Deferred Type: Standards Track Content-Type: text/x-rst
Requires: 3107, 3115, 3119 Created: 28-Apr-2007 Post-History:
30-Apr-2007 Replaces: 245, 246

Deferred

See https://mail.python.org/pipermail/python-3000/2007-July/008784.html.

Abstract

This PEP proposes a new standard library module, overloading, to provide
generic programming features including dynamic overloading (aka generic
functions), interfaces, adaptation, method combining (ala CLOS and
AspectJ), and simple forms of aspect-oriented programming (AOP).

The proposed API is also open to extension; that is, it will be possible
for library developers to implement their own specialized interface
types, generic function dispatchers, method combination algorithms,
etc., and those extensions will be treated as first-class citizens by
the proposed API.

The API will be implemented in pure Python with no C, but may have some
dependency on CPython-specific features such as sys._getframe and the
func_code attribute of functions. It is expected that e.g. Jython and
IronPython will have other ways of implementing similar functionality
(perhaps using Java or C#).

Rationale and Goals

Python has always provided a variety of built-in and standard-library
generic functions, such as len(), iter(), pprint.pprint(), and most of
the functions in the operator module. However, it currently:

1.  does not have a simple or straightforward way for developers to
    create new generic functions,
2.  does not have a standard way for methods to be added to existing
    generic functions (i.e., some are added using registration
    functions, others require defining __special__ methods, possibly by
    monkeypatching), and
3.  does not allow dispatching on multiple argument types (except in a
    limited form for arithmetic operators, where "right-hand" (__r*__)
    methods can be used to do two-argument dispatch.

In addition, it is currently a common anti-pattern for Python code to
inspect the types of received arguments, in order to decide what to do
with the objects. For example, code may wish to accept either an object
of some type, or a sequence of objects of that type.

Currently, the "obvious way" to do this is by type inspection, but this
is brittle and closed to extension. A developer using an already-written
library may be unable to change how their objects are treated by such
code, especially if the objects they are using were created by a third
party.

Therefore, this PEP proposes a standard library module to address these,
and related issues, using decorators and argument annotations (PEP
3107). The primary features to be provided are:

-   a dynamic overloading facility, similar to the static overloading
    found in languages such as Java and C++, but including optional
    method combination features as found in CLOS and AspectJ.
-   a simple "interfaces and adaptation" library inspired by Haskell's
    typeclasses (but more dynamic, and without any static
    type-checking), with an extension API to allow registering
    user-defined interface types such as those found in PyProtocols and
    Zope.
-   a simple "aspect" implementation to make it easy to create stateful
    adapters and to do other stateful AOP.

These features are to be provided in such a way that extended
implementations can be created and used. For example, it should be
possible for libraries to define new dispatching criteria for generic
functions, and new kinds of interfaces, and use them in place of the
predefined features. For example, it should be possible to use a
zope.interface interface object to specify the desired type of a
function argument, as long as the zope.interface package registered
itself correctly (or a third party did the registration).

In this way, the proposed API simply offers a uniform way of accessing
the functionality within its scope, rather than prescribing a single
implementation to be used for all libraries, frameworks, and
applications.

User API

The overloading API will be implemented as a single module, named
overloading, providing the following features:

Overloading/Generic Functions

The @overload decorator allows you to define alternate implementations
of a function, specialized by argument type(s). A function with the same
name must already exist in the local namespace. The existing function is
modified in-place by the decorator to add the new implementation, and
the modified function is returned by the decorator. Thus, the following
code:

    from overloading import overload
    from collections import Iterable

    def flatten(ob):
        """Flatten an object to its component iterables"""
        yield ob

    @overload
    def flatten(ob: Iterable):
        for o in ob:
            for ob in flatten(o):
                yield ob

    @overload
    def flatten(ob: basestring):
        yield ob

creates a single flatten() function whose implementation roughly equates
to:

    def flatten(ob):
        if isinstance(ob, basestring) or not isinstance(ob, Iterable):
            yield ob
        else:
            for o in ob:
                for ob in flatten(o):
                    yield ob

except that the flatten() function defined by overloading remains open
to extension by adding more overloads, while the hardcoded version
cannot be extended.

For example, if someone wants to use flatten() with a string-like type
that doesn't subclass basestring, they would be out of luck with the
second implementation. With the overloaded implementation, however, they
can either write this:

    @overload
    def flatten(ob: MyString):
        yield ob

or this (to avoid copying the implementation):

    from overloading import RuleSet
    RuleSet(flatten).copy_rules((basestring,), (MyString,))

(Note also that, although PEP 3119 proposes that it should be possible
for abstract base classes like Iterable to allow classes like MyString
to claim subclass-hood, such a claim is global, throughout the
application. In contrast, adding a specific overload or copying a rule
is specific to an individual function, and therefore less likely to have
undesired side effects.)

@overload vs. @when

The @overload decorator is a common-case shorthand for the more general
@when decorator. It allows you to leave out the name of the function you
are overloading, at the expense of requiring the target function to be
in the local namespace. It also doesn't support adding additional
criteria besides the ones specified via argument annotations. The
following function definitions have identical effects, except for name
binding side-effects (which will be described below):

    from overloading import when

    @overload
    def flatten(ob: basestring):
        yield ob

    @when(flatten)
    def flatten(ob: basestring):
        yield ob

    @when(flatten)
    def flatten_basestring(ob: basestring):
        yield ob

    @when(flatten, (basestring,))
    def flatten_basestring(ob):
        yield ob

The first definition above will bind flatten to whatever it was
previously bound to. The second will do the same, if it was already
bound to the when decorator's first argument. If flatten is unbound or
bound to something else, it will be rebound to the function definition
as given. The last two definitions above will always bind
flatten_basestring to the function definition as given.

Using this approach allows you to both give a method a descriptive name
(often useful in tracebacks!) and to reuse the method later.

Except as otherwise specified, all overloading decorators have the same
signature and binding rules as @when. They accept a function and an
optional "predicate" object.

The default predicate implementation is a tuple of types with positional
matching to the overloaded function's arguments. However, an arbitrary
number of other kinds of predicates can be created and registered using
the Extension API, and will then be usable with @when and other
decorators created by this module (like @before, @after, and @around).

Method Combination and Overriding

When an overloaded function is invoked, the implementation with the
signature that most specifically matches the calling arguments is the
one used. If no implementation matches, a NoApplicableMethods error is
raised. If more than one implementation matches, but none of the
signatures are more specific than the others, an AmbiguousMethods error
is raised.

For example, the following pair of implementations are ambiguous, if the
foo() function is ever called with two integer arguments, because both
signatures would apply, but neither signature is more specific than the
other (i.e., neither implies the other):

    def foo(bar:int, baz:object):
        pass

    @overload
    def foo(bar:object, baz:int):
        pass

In contrast, the following pair of implementations can never be
ambiguous, because one signature always implies the other; the int/int
signature is more specific than the object/object signature:

    def foo(bar:object, baz:object):
        pass

    @overload
    def foo(bar:int, baz:int):
        pass

A signature S1 implies another signature S2, if whenever S1 would apply,
S2 would also. A signature S1 is "more specific" than another signature
S2, if S1 implies S2, but S2 does not imply S1.

Although the examples above have all used concrete or abstract types as
argument annotations, there is no requirement that the annotations be
such. They can also be "interface" objects (discussed in the Interfaces
and Adaptation section), including user-defined interface types. (They
can also be other objects whose types are appropriately registered via
the Extension API.)

Proceeding to the "Next" Method

If the first parameter of an overloaded function is named __proceed__,
it will be passed a callable representing the next most-specific method.
For example, this code:

    def foo(bar:object, baz:object):
        print "got objects!"

    @overload
    def foo(__proceed__, bar:int, baz:int):
        print "got integers!"
        return __proceed__(bar, baz)

Will print "got integers!" followed by "got objects!".

If there is no next most-specific method, __proceed__ will be bound to a
NoApplicableMethods instance. When called, a new NoApplicableMethods
instance will be raised, with the arguments passed to the first
instance.

Similarly, if the next most-specific methods have ambiguous precedence
with respect to each other, __proceed__ will be bound to an
AmbiguousMethods instance, and if called, it will raise a new instance.

Thus, a method can either check if __proceed__ is an error instance, or
simply invoke it. The NoApplicableMethods and AmbiguousMethods error
classes have a common DispatchError base class, so
isinstance(__proceed__, overloading.DispatchError) is sufficient to
identify whether __proceed__ can be safely called.

(Implementation note: using a magic argument name like __proceed__ could
potentially be replaced by a magic function that would be called to
obtain the next method. A magic function, however, would degrade
performance and might be more difficult to implement on non-CPython
platforms. Method chaining via magic argument names, however, can be
efficiently implemented on any Python platform that supports creating
bound methods from functions -- one simply recursively binds each
function to be chained, using the following function or error as the
im_self of the bound method.)

"Before" and "After" Methods

In addition to the simple next-method chaining shown above, it is
sometimes useful to have other ways of combining methods. For example,
the "observer pattern" can sometimes be implemented by adding extra
methods to a function, that execute before or after the normal
implementation.

To support these use cases, the overloading module will supply @before,
@after, and @around decorators, that roughly correspond to the same
types of methods in the Common Lisp Object System (CLOS), or the
corresponding "advice" types in AspectJ.

Like @when, all of these decorators must be passed the function to be
overloaded, and can optionally accept a predicate as well:

    from overloading import before, after

    def begin_transaction(db):
        print "Beginning the actual transaction"

    @before(begin_transaction)
    def check_single_access(db: SingletonDB):
        if db.inuse:
            raise TransactionError("Database already in use")

    @after(begin_transaction)
    def start_logging(db: LoggableDB):
        db.set_log_level(VERBOSE)

@before and @after methods are invoked either before or after the main
function body, and are never considered ambiguous. That is, it will not
cause any errors to have multiple "before" or "after" methods with
identical or overlapping signatures. Ambiguities are resolved using the
order in which the methods were added to the target function.

"Before" methods are invoked most-specific method first, with ambiguous
methods being executed in the order they were added. All "before"
methods are called before any of the function's "primary" methods (i.e.
normal @overload methods) are executed.

"After" methods are invoked in the reverse order, after all of the
function's "primary" methods are executed. That is, they are executed
least-specific methods first, with ambiguous methods being executed in
the reverse of the order in which they were added.

The return values of both "before" and "after" methods are ignored, and
any uncaught exceptions raised by any methods (primary or other)
immediately end the dispatching process. "Before" and "after" methods
cannot have __proceed__ arguments, as they are not responsible for
calling any other methods. They are simply called as a notification
before or after the primary methods.

Thus, "before" and "after" methods can be used to check or establish
preconditions (e.g. by raising an error if the conditions aren't met) or
to ensure postconditions, without needing to duplicate any existing
functionality.

"Around" Methods

The @around decorator declares a method as an "around" method. "Around"
methods are much like primary methods, except that the least-specific
"around" method has higher precedence than the most-specific "before"
method.

Unlike "before" and "after" methods, however, "Around" methods are
responsible for calling their __proceed__ argument, in order to continue
the invocation process. "Around" methods are usually used to transform
input arguments or return values, or to wrap specific cases with special
error handling or try/finally conditions, e.g.:

    from overloading import around

    @around(commit_transaction)
    def lock_while_committing(__proceed__, db: SingletonDB):
        with db.global_lock:
            return __proceed__(db)

They can also be used to replace the normal handling for a specific
case, by not invoking the __proceed__ function.

The __proceed__ given to an "around" method will either be the next
applicable "around" method, a DispatchError instance, or a synthetic
method object that will call all the "before" methods, followed by the
primary method chain, followed by all the "after" methods, and return
the result from the primary method chain.

Thus, just as with normal methods, __proceed__ can be checked for
DispatchError-ness, or simply invoked. The "around" method should return
the value returned by __proceed__, unless of course it wishes to modify
or replace it with a different return value for the function as a whole.

Custom Combinations

The decorators described above (@overload, @when, @before, @after, and
@around) collectively implement what in CLOS is called the "standard
method combination" -- the most common patterns used in combining
methods.

Sometimes, however, an application or library may have use for a more
sophisticated type of method combination. For example, if you would like
to have "discount" methods that return a percentage off, to be
subtracted from the value returned by the primary method(s), you might
write something like this:

    from overloading import always_overrides, merge_by_default
    from overloading import Around, Before, After, Method, MethodList

    class Discount(MethodList):
        """Apply return values as discounts"""

        def __call__(self, *args, **kw):
            retval = self.tail(*args, **kw)
            for sig, body in self.sorted():
                retval -= retval * body(*args, **kw)
            return retval

    # merge discounts by priority
    merge_by_default(Discount)

    # discounts have precedence over before/after/primary methods
    always_overrides(Discount, Before)
    always_overrides(Discount, After)
    always_overrides(Discount, Method)

    # but not over "around" methods
    always_overrides(Around, Discount)

    # Make a decorator called "discount" that works just like the
    # standard decorators...
    discount = Discount.make_decorator('discount')

    # and now let's use it...
    def price(product):
        return product.list_price

    @discount(price)
    def ten_percent_off_shoes(product: Shoe)
        return Decimal('0.1')

Similar techniques can be used to implement a wide variety of CLOS-style
method qualifiers and combination rules. The process of creating custom
method combination objects and their corresponding decorators is
described in more detail under the Extension API section.

Note, by the way, that the @discount decorator shown will work correctly
with any new predicates defined by other code. For example, if
zope.interface were to register its interface types to work correctly as
argument annotations, you would be able to specify discounts on the
basis of its interface types, not just classes or overloading-defined
interface types.

Similarly, if a library like RuleDispatch or PEAK-Rules were to register
an appropriate predicate implementation and dispatch engine, one would
then be able to use those predicates for discounts as well, e.g.:

    from somewhere import Pred  # some predicate implementation

    @discount(
        price,
        Pred("isinstance(product,Shoe) and"
             " product.material.name=='Blue Suede'")
    )
    def forty_off_blue_suede_shoes(product):
        return Decimal('0.4')

The process of defining custom predicate types and dispatching engines
is also described in more detail under the Extension API section.

Overloading Inside Classes

All of the decorators above have a special additional behavior when they
are directly invoked within a class body: the first parameter (other
than __proceed__, if present) of the decorated function will be treated
as though it had an annotation equal to the class in which it was
defined.

That is, this code:

    class And(object):
        # ...
        @when(get_conjuncts)
        def __conjuncts(self):
            return self.conjuncts

produces the same effect as this (apart from the existence of a private
method):

    class And(object):
        # ...

    @when(get_conjuncts)
    def get_conjuncts_of_and(ob: And):
        return ob.conjuncts

This behavior is both a convenience enhancement when defining lots of
methods, and a requirement for safely distinguishing multi-argument
overloads in subclasses. Consider, for example, the following code:

    class A(object):
        def foo(self, ob):
            print "got an object"

        @overload
        def foo(__proceed__, self, ob:Iterable):
            print "it's iterable!"
            return __proceed__(self, ob)


    class B(A):
        foo = A.foo     # foo must be defined in local namespace

        @overload
        def foo(__proceed__, self, ob:Iterable):
            print "B got an iterable!"
            return __proceed__(self, ob)

Due to the implicit class rule, calling B().foo([]) will print "B got an
iterable!" followed by "it's iterable!", and finally, "got an object",
while A().foo([]) would print only the messages defined in A.

Conversely, without the implicit class rule, the two "Iterable" methods
would have the exact same applicability conditions, so calling either
A().foo([]) or B().foo([]) would result in an AmbiguousMethods error.

It is currently an open issue to determine the best way to implement
this rule in Python 3.0. Under Python 2.x, a class' metaclass was not
chosen until the end of the class body, which means that decorators
could insert a custom metaclass to do processing of this sort. (This is
how RuleDispatch, for example, implements the implicit class rule.)

PEP 3115, however, requires that a class' metaclass be determined before
the class body has executed, making it impossible to use this technique
for class decoration any more.

At this writing, discussion on this issue is ongoing.

Interfaces and Adaptation

The overloading module provides a simple implementation of interfaces
and adaptation. The following example defines an IStack interface, and
declares that list objects support it:

    from overloading import abstract, Interface

    class IStack(Interface):
        @abstract
        def push(self, ob)
            """Push 'ob' onto the stack"""

        @abstract
        def pop(self):
            """Pop a value and return it"""


    when(IStack.push, (list, object))(list.append)
    when(IStack.pop, (list,))(list.pop)

    mylist = []
    mystack = IStack(mylist)
    mystack.push(42)
    assert mystack.pop()==42

The Interface class is a kind of "universal adapter". It accepts a
single argument: an object to adapt. It then binds all its methods to
the target object, in place of itself. Thus, calling mystack.push(42) is
the same as calling IStack.push(mylist, 42).

The @abstract decorator marks a function as being abstract: i.e., having
no implementation. If an @abstract function is called, it raises
NoApplicableMethods. To become executable, overloaded methods must be
added using the techniques previously described. (That is, methods can
be added using @when, @before, @after, @around, or any custom method
combination decorators.)

In the example above, the list.append method is added as a method for
IStack.push() when its arguments are a list and an arbitrary object.
Thus, IStack.push(mylist, 42) is translated to list.append(mylist, 42),
thereby implementing the desired operation.

Abstract and Concrete Methods

Note, by the way, that the @abstract decorator is not limited to use in
interface definitions; it can be used anywhere that you wish to create
an "empty" generic function that initially has no methods. In
particular, it need not be used inside a class.

Also note that interface methods need not be abstract; one could, for
example, write an interface like this:

    class IWriteMapping(Interface):
        @abstract
        def __setitem__(self, key, value):
            """This has to be implemented"""

        def update(self, other:IReadMapping):
            for k, v in IReadMapping(other).items():
                self[k] = v

As long as __setitem__ is defined for some type, the above interface
will provide a usable update() implementation. However, if some specific
type (or pair of types) has a more efficient way of handling update()
operations, an appropriate overload can still be registered for use in
that case.

Subclassing and Re-assembly

Interfaces can be subclassed:

    class ISizedStack(IStack):
        @abstract
        def __len__(self):
            """Return the number of items on the stack"""

    # define __len__ support for ISizedStack
    when(ISizedStack.__len__, (list,))(list.__len__)

Or assembled by combining functions from existing interfaces:

    class Sizable(Interface):
        __len__ = ISizedStack.__len__

    # list now implements Sizable as well as ISizedStack, without
    # making any new declarations!

A class can be considered to "adapt to" an interface at a given point in
time, if no method defined in the interface is guaranteed to raise a
NoApplicableMethods error if invoked on an instance of that class at
that point in time.

In normal usage, however, it is "easier to ask forgiveness than
permission". That is, it is easier to simply use an interface on an
object by adapting it to the interface (e.g. IStack(mylist)) or invoking
interface methods directly (e.g. IStack.push(mylist, 42)), than to try
to figure out whether the object is adaptable to (or directly
implements) the interface.

Implementing an Interface in a Class

It is possible to declare that a class directly implements an interface,
using the declare_implementation() function:

    from overloading import declare_implementation

    class Stack(object):
        def __init__(self):
            self.data = []
        def push(self, ob):
            self.data.append(ob)
        def pop(self):
            return self.data.pop()

    declare_implementation(IStack, Stack)

The declare_implementation() call above is roughly equivalent to the
following steps:

    when(IStack.push, (Stack,object))(lambda self, ob: self.push(ob))
    when(IStack.pop, (Stack,))(lambda self, ob: self.pop())

That is, calling IStack.push() or IStack.pop() on an instance of any
subclass of Stack, will simply delegate to the actual push() or pop()
methods thereof.

For the sake of efficiency, calling IStack(s) where s is an instance of
Stack, may return s rather than an IStack adapter. (Note that calling
IStack(x) where x is already an IStack adapter will always return x
unchanged; this is an additional optimization allowed in cases where the
adaptee is known to directly implement the interface, without
adaptation.)

For convenience, it may be useful to declare implementations in the
class header, e.g.:

    class Stack(metaclass=Implementer, implements=IStack):
        ...

Instead of calling declare_implementation() after the end of the suite.

Interfaces as Type Specifiers

Interface subclasses can be used as argument annotations to indicate
what type of objects are acceptable to an overload, e.g.:

    @overload
    def traverse(g: IGraph, s: IStack):
        g = IGraph(g)
        s = IStack(s)
        # etc....

Note, however, that the actual arguments are not changed or adapted in
any way by the mere use of an interface as a type specifier. You must
explicitly cast the objects to the appropriate interface, as shown
above.

Note, however, that other patterns of interface use are possible. For
example, other interface implementations might not support adaptation,
or might require that function arguments already be adapted to the
specified interface. So the exact semantics of using an interface as a
type specifier are dependent on the interface objects you actually use.

For the interface objects defined by this PEP, however, the semantics
are as described above. An interface I1 is considered "more specific"
than another interface I2, if the set of descriptors in I1's inheritance
hierarchy are a proper superset of the descriptors in I2's inheritance
hierarchy.

So, for example, ISizedStack is more specific than both ISizable and
ISizedStack, irrespective of the inheritance relationships between these
interfaces. It is purely a question of what operations are included
within those interfaces -- and the names of the operations are
unimportant.

Interfaces (at least the ones provided by overloading) are always
considered less-specific than concrete classes. Other interface
implementations can decide on their own specificity rules, both between
interfaces and other interfaces, and between interfaces and classes.

Non-Method Attributes in Interfaces

The Interface implementation actually treats all attributes and methods
(i.e. descriptors) in the same way: their __get__ (and __set__ and
__delete__, if present) methods are called with the wrapped (adapted)
object as "self". For functions, this has the effect of creating a bound
method linking the generic function to the wrapped object.

For non-function attributes, it may be easiest to specify them using the
property built-in, and the corresponding fget, fset, and fdel
attributes:

    class ILength(Interface):
        @property
        @abstract
        def length(self):
            """Read-only length attribute"""

    # ILength(aList).length == list.__len__(aList)
    when(ILength.length.fget, (list,))(list.__len__)

Alternatively, methods such as _get_foo() and _set_foo() may be defined
as part of the interface, and the property defined in terms of those
methods, but this is a bit more difficult for users to implement
correctly when creating a class that directly implements the interface,
as they would then need to match all the individual method names, not
just the name of the property or attribute.

Aspects

The adaptation system described above assumes that adapters are
"stateless", which is to say that adapters have no attributes or state
apart from that of the adapted object. This follows the
"typeclass/instance" model of Haskell, and the concept of "pure" (i.e.,
transitively composable) adapters.

However, there are occasionally cases where, to provide a complete
implementation of some interface, some sort of additional state is
required.

One possibility of course, would be to attach monkeypatched "private"
attributes to the adaptee. But this is subject to name collisions, and
complicates the process of initialization (since any code using these
attributes has to check for their existence and initialize them if
necessary). It also doesn't work on objects that don't have a __dict__
attribute.

So the Aspect class is provided to make it easy to attach extra
information to objects that either:

1.  have a __dict__ attribute (so aspect instances can be stored in it,
    keyed by aspect class),
2.  support weak referencing (so aspect instances can be managed using a
    global but thread-safe weak-reference dictionary), or
3.  implement or can be adapt to the overloading.IAspectOwner interface
    (technically, #1 or #2 imply this).

Subclassing Aspect creates an adapter class whose state is tied to the
life of the adapted object.

For example, suppose you would like to count all the times a certain
method is called on instances of Target (a classic AOP example). You
might do something like:

    from overloading import Aspect

    class Count(Aspect):
        count = 0

    @after(Target.some_method)
    def count_after_call(self:Target, *args, **kw):
        Count(self).count += 1

The above code will keep track of the number of times that
Target.some_method() is successfully called on an instance of Target
(i.e., it will not count errors unless they occur in a more-specific
"after" method). Other code can then access the count using
Count(someTarget).count.

Aspect instances can of course have __init__ methods, to initialize any
data structures. They can use either __slots__ or dictionary-based
attributes for storage.

While this facility is rather primitive compared to a full-featured AOP
tool like AspectJ, persons who wish to build pointcut libraries or other
AspectJ-like features can certainly use Aspect objects and
method-combination decorators as a base for building more expressive AOP
tools.

XXX spec out full aspect API, including keys, N-to-1 aspects, manual

    attach/detach/delete of aspect instances, and the IAspectOwner
    interface.

Extension API

TODO: explain how all of these work

implies(o1, o2)

declare_implementation(iface, class)

predicate_signatures(ob)

parse_rule(ruleset, body, predicate, actiontype, localdict, globaldict)

combine_actions(a1, a2)

rules_for(f)

Rule objects

ActionDef objects

RuleSet objects

Method objects

MethodList objects

IAspectOwner

Overloading Usage Patterns

In discussion on the Python-3000 list, the proposed feature of allowing
arbitrary functions to be overloaded has been somewhat controversial,
with some people expressing concern that this would make programs more
difficult to understand.

The general thrust of this argument is that one cannot rely on what a
function does, if it can be changed from anywhere in the program at any
time. Even though in principle this can already happen through
monkeypatching or code substitution, it is considered poor practice to
do so.

However, providing support for overloading any function (or so the
argument goes), is implicitly blessing such changes as being an
acceptable practice.

This argument appears to make sense in theory, but it is almost entirely
mooted in practice for two reasons.

First, people are generally not perverse, defining a function to do one
thing in one place, and then summarily defining it to do the opposite
somewhere else! The principal reasons to extend the behavior of a
function that has not been specifically made generic are to:

-   Add special cases not contemplated by the original function's
    author, such as support for additional types.
-   Be notified of an action in order to cause some related operation to
    be performed, either before the original operation is performed,
    after it, or both. This can include general-purpose operations like
    adding logging, timing, or tracing, as well as application-specific
    behavior.

None of these reasons for adding overloads imply any change to the
intended default or overall behavior of the existing function, however.
Just as a base class method may be overridden by a subclass for these
same two reasons, so too may a function be overloaded to provide for
such enhancements.

In other words, universal overloading does not equal arbitrary
overloading, in the sense that we need not expect people to randomly
redefine the behavior of existing functions in illogical or
unpredictable ways. If they did so, it would be no less of a bad
practice than any other way of writing illogical or unpredictable code!

However, to distinguish bad practice from good, it is perhaps necessary
to clarify further what good practice for defining overloads is. And
that brings us to the second reason why generic functions do not
necessarily make programs harder to understand: overloading patterns in
actual programs tend to follow very predictable patterns. (Both in
Python and in languages that have no non-generic functions.)

If a module is defining a new generic operation, it will usually also
define any required overloads for existing types in the same place.
Likewise, if a module is defining a new type, then it will usually
define overloads there for any generic functions that it knows or cares
about.

As a result, the vast majority of overloads can be found adjacent to
either the function being overloaded, or to a newly-defined type for
which the overload is adding support. Thus, overloads are
highly-discoverable in the common case, as you are either looking at the
function or the type, or both.

It is only in rather infrequent cases that one will have overloads in a
module that contains neither the function nor the type(s) for which the
overload is added. This would be the case if, say, a third-party created
a bridge of support between one library's types and another library's
generic function(s). In such a case, however, best practice suggests
prominently advertising this, especially by way of the module name.

For example, PyProtocols defines such bridge support for working with
Zope interfaces and legacy Twisted interfaces, using modules called
protocols.twisted_support and protocols.zope_support. (These bridges are
done with interface adapters, rather than generic functions, but the
basic principle is the same.)

In short, understanding programs in the presence of universal
overloading need not be any more difficult, given that the vast majority
of overloads will either be adjacent to a function, or the definition of
a type that is passed to that function.

And, in the absence of incompetence or deliberate intention to be
obscure, the few overloads that are not adjacent to the relevant type(s)
or function(s), will generally not need to be understood or known about
outside the scope where those overloads are defined. (Except in the
"support modules" case, where best practice suggests naming them
accordingly.)

Implementation Notes

Most of the functionality described in this PEP is already implemented
in the in-development version of the PEAK-Rules framework. In
particular, the basic overloading and method combination framework
(minus the @overload decorator) already exists there. The implementation
of all of these features in peak.rules.core is 656 lines of Python at
this writing.

peak.rules.core currently relies on the DecoratorTools and
BytecodeAssembler modules, but both of these dependencies can be
replaced, as DecoratorTools is used mainly for Python 2.3 compatibility
and to implement structure types (which can be done with named tuples in
later versions of Python). The use of BytecodeAssembler can be replaced
using an "exec" or "compile" workaround, given a reasonable effort. (It
would be easier to do this if the func_closure attribute of function
objects was writable.)

The Interface class has been previously prototyped, but is not included
in PEAK-Rules at the present time.

The "implicit class rule" has previously been implemented in the
RuleDispatch library. However, it relies on the __metaclass__ hook that
is currently eliminated in PEP 3115.

I don't currently know how to make @overload play nicely with
classmethod and staticmethod in class bodies. It's not really clear if
it needs to, however.

Copyright

This document has been placed in the public domain.



  Local Variables: mode: indented-text indent-tabs-mode: nil
  sentence-end-double-space: t fill-column: 70 coding: utf-8 End: