PEP: 3104 Title: Access to Names in Outer Scopes Author: Ka-Ping Yee
<ping@zesty.ca> Status: Final Type: Standards Track Content-Type:
text/x-rst Created: 12-Oct-2006 Python-Version: 3.0 Post-History:

Abstract

In most languages that support nested scopes, code can refer to or
rebind (assign to) any name in the nearest enclosing scope. Currently,
Python code can refer to a name in any enclosing scope, but it can only
rebind names in two scopes: the local scope (by simple assignment) or
the module-global scope (using a global declaration).

This limitation has been raised many times on the Python-Dev mailing
list and elsewhere, and has led to extended discussion and many
proposals for ways to remove this limitation. This PEP summarizes the
various alternatives that have been suggested, together with advantages
and disadvantages that have been mentioned for each.

Rationale

Before version 2.1, Python's treatment of scopes resembled that of
standard C: within a file there were only two levels of scope, global
and local. In C, this is a natural consequence of the fact that function
definitions cannot be nested. But in Python, though functions are
usually defined at the top level, a function definition can be executed
anywhere. This gave Python the syntactic appearance of nested scoping
without the semantics, and yielded inconsistencies that were surprising
to some programmers -- for example, a recursive function that worked at
the top level would cease to work when moved inside another function,
because the recursive function's own name would no longer be visible in
its body's scope. This violates the intuition that a function should
behave consistently when placed in different contexts. Here's an
example:

    def enclosing_function():
        def factorial(n):
            if n < 2:
                return 1
            return n * factorial(n - 1)  # fails with NameError
        print factorial(5)

Python 2.1 moved closer to static nested scoping by making visible the
names bound in all enclosing scopes (see PEP 227). This change makes the
above code example work as expected. However, because any assignment to
a name implicitly declares that name to be local, it is impossible to
rebind a name in an outer scope (except when a global declaration forces
the name to be global). Thus, the following code, intended to display a
number that can be incremented and decremented by clicking buttons,
doesn't work as someone familiar with lexical scoping might expect:

    def make_scoreboard(frame, score=0):
        label = Label(frame)
        label.pack()
        for i in [-10, -1, 1, 10]:
            def increment(step=i):
                score = score + step  # fails with UnboundLocalError
                label['text'] = score
            button = Button(frame, text='%+d' % i, command=increment)
            button.pack()
        return label

Python syntax doesn't provide a way to indicate that the name score
mentioned in increment refers to the variable score bound in
make_scoreboard, not a local variable in increment. Users and developers
of Python have expressed an interest in removing this limitation so that
Python can have the full flexibility of the Algol-style scoping model
that is now standard in many programming languages, including
JavaScript, Perl, Ruby, Scheme, Smalltalk, C with GNU extensions, and C#
2.0.

It has been argued that such a feature isn't necessary, because a
rebindable outer variable can be simulated by wrapping it in a mutable
object:

    class Namespace:
        pass

    def make_scoreboard(frame, score=0):
        ns = Namespace()
        ns.score = 0
        label = Label(frame)
        label.pack()
        for i in [-10, -1, 1, 10]:
            def increment(step=i):
                ns.score = ns.score + step
                label['text'] = ns.score
            button = Button(frame, text='%+d' % i, command=increment)
            button.pack()
        return label

However, this workaround only highlights the shortcomings of existing
scopes: the purpose of a function is to encapsulate code in its own
namespace, so it seems unfortunate that the programmer should have to
create additional namespaces to make up for missing functionality in the
existing local scopes, and then have to decide whether each name should
reside in the real scope or the simulated scope.

Another common objection is that the desired functionality can be
written as a class instead, albeit somewhat more verbosely. One rebuttal
to this objection is that the existence of a different implementation
style is not a reason to leave a supported programming construct (nested
scopes) functionally incomplete. Python is sometimes called a
"multi-paradigm language" because it derives so much strength, practical
flexibility, and pedagogical power from its support and graceful
integration of multiple programming paradigms.

A proposal for scoping syntax appeared on Python-Dev as far back as
1994[1], long before PEP 227's support for nested scopes was adopted. At
the time, Guido's response was:

  This is dangerously close to introducing CSNS [classic static nested
  scopes]. If you were to do so, your proposed semantics of scoped seem
  alright. I still think there is not enough need for CSNS to warrant
  this kind of construct ...

After PEP 227, the "outer name rebinding discussion" has reappeared on
Python-Dev enough times that it has become a familiar event, having
recurred in its present form since at least 2003[2]. Although none of
the language changes proposed in these discussions have yet been
adopted, Guido has acknowledged that a language change is worth
considering[3].

Other Languages

To provide some background, this section describes how some other
languages handle nested scopes and rebinding.

JavaScript, Perl, Scheme, Smalltalk, GNU C, C# 2.0

These languages use variable declarations to indicate scope. In
JavaScript, a lexically scoped variable is declared with the var
keyword; undeclared variable names are assumed to be global. In Perl, a
lexically scoped variable is declared with the my keyword; undeclared
variable names are assumed to be global. In Scheme, all variables must
be declared (with define or let, or as formal parameters). In Smalltalk,
any block can begin by declaring a list of local variable names between
vertical bars. C and C# require type declarations for all variables. For
all these cases, the variable belongs to the scope containing the
declaration.

Ruby (as of 1.8)

Ruby is an instructive example because it appears to be the only other
currently popular language that, like Python, tries to support
statically nested scopes without requiring variable declarations, and
thus has to come up with an unusual solution. Functions in Ruby can
contain other function definitions, and they can also contain code
blocks enclosed in curly braces. Blocks have access to outer variables,
but nested functions do not. Within a block, an assignment to a name
implies a declaration of a local variable only if it would not shadow a
name already bound in an outer scope; otherwise assignment is
interpreted as rebinding of the outer name. Ruby's scoping syntax and
rules have also been debated at great length, and changes seem likely in
Ruby 2.0[4].

Overview of Proposals

There have been many different proposals on Python-Dev for ways to
rebind names in outer scopes. They all fall into two categories: new
syntax in the scope where the name is bound, or new syntax in the scope
where the name is used.

New Syntax in the Binding (Outer) Scope

Scope Override Declaration

The proposals in this category all suggest a new kind of declaration
statement similar to JavaScript's var. A few possible keywords have been
proposed for this purpose:

-   scope x[5]
-   var x[6][7]
-   my x[8]

In all these proposals, a declaration such as var x in a particular
scope S would cause all references to x in scopes nested within S to
refer to the x bound in S.

The primary objection to this category of proposals is that the meaning
of a function definition would become context-sensitive. Moving a
function definition inside some other block could cause any of the local
name references in the function to become nonlocal, due to declarations
in the enclosing block. For blocks in Ruby 1.8, this is actually the
case; in the following example, the two setters have different effects
even though they look identical:

    setter1 = proc { | x | y = x }      # y is local here
    y = 13
    setter2 = proc { | x | y = x }      # y is nonlocal here
    setter1.call(99)
    puts y                              # prints 13
    setter2.call(77)
    puts y                              # prints 77

Note that although this proposal resembles declarations in JavaScript
and Perl, the effect on the language is different because in those
languages undeclared variables are global by default, whereas in Python
undeclared variables are local by default. Thus, moving a function
inside some other block in JavaScript or Perl can only reduce the scope
of a previously global name reference, whereas in Python with this
proposal, it could expand the scope of a previously local name
reference.

Required Variable Declaration

A more radical proposal[9] suggests removing Python's scope-guessing
convention altogether and requiring that all names be declared in the
scope where they are to be bound, much like Scheme. With this proposal,
var x = 3 would both declare x to belong to the local scope and bind it,
where as x = 3 would rebind the existing visible x. In a context without
an enclosing scope containing a var x declaration, the statement x = 3
would be statically determined to be illegal.

This proposal yields a simple and consistent model, but it would be
incompatible with all existing Python code.

New Syntax in the Referring (Inner) Scope

There are three kinds of proposals in this category.

Outer Reference Expression

This type of proposal suggests a new way of referring to a variable in
an outer scope when using the variable in an expression. One syntax that
has been suggested for this is .x[10], which would refer to x without
creating a local binding for it. A concern with this proposal is that in
many contexts x and .x could be used interchangeably, which would
confuse the reader[11]. A closely related idea is to use multiple dots
to specify the number of scope levels to ascend[12], but most consider
this too error-prone [13].

Rebinding Operator

This proposal suggests a new assignment-like operator that rebinds a
name without declaring the name to be local[14]. Whereas the statement
x = 3 both declares x a local variable and binds it to 3, the statement
x := 3 would change the existing binding of x without declaring it
local.

This is a simple solution, but according to PEP 3099 it has been
rejected (perhaps because it would be too easy to miss or to confuse
with =).

Scope Override Declaration

The proposals in this category suggest a new kind of declaration
statement in the inner scope that prevents a name from becoming local.
This statement would be similar in nature to the global statement, but
instead of making the name refer to a binding in the top module-level
scope, it would make the name refer to the binding in the nearest
enclosing scope.

This approach is attractive due to its parallel with a familiar Python
construct, and because it retains context-independence for function
definitions.

This approach also has advantages from a security and debugging
perspective. The resulting Python would not only match the functionality
of other nested-scope languages but would do so with a syntax that is
arguably even better for defensive programming. In most other languages,
a declaration contracts the scope of an existing name, so inadvertently
omitting the declaration could yield farther-reaching (i.e. more
dangerous) effects than expected. In Python with this proposal, the
extra effort of adding the declaration is aligned with the increased
risk of non-local effects (i.e. the path of least resistance is the
safer path).

Many spellings have been suggested for such a declaration:

-   scoped x[15]
-   global x in f[16] (explicitly specify which scope)
-   free x[17]
-   outer x[18]
-   use x[19]
-   global x[20] (change the meaning of global)
-   nonlocal x[21]
-   global x outer[22]
-   global in x[23]
-   not global x[24]
-   extern x[25]
-   ref x[26]
-   refer x[27]
-   share x[28]
-   sharing x[29]
-   common x[30]
-   using x[31]
-   borrow x[32]
-   reuse x[33]
-   scope f x[34] (explicitly specify which scope)

The most commonly discussed choices appear to be outer, global, and
nonlocal. outer is already used as both a variable name and an attribute
name in the standard library. The word global has a conflicting meaning,
because "global variable" is generally understood to mean a variable
with top-level scope[35]. In C, the keyword extern means that a name
refers to a variable in a different compilation unit. While nonlocal is
a bit long and less pleasant-sounding than some of the other options, it
does have precisely the correct meaning: it declares a name not local.

Proposed Solution

The solution proposed by this PEP is to add a scope override declaration
in the referring (inner) scope. Guido has expressed a preference for
this category of solution on Python-Dev[36] and has shown approval for
nonlocal as the keyword[37].

The proposed declaration:

    nonlocal x

prevents x from becoming a local name in the current scope. All
occurrences of x in the current scope will refer to the x bound in an
outer enclosing scope. As with global, multiple names are permitted:

    nonlocal x, y, z

If there is no pre-existing binding in an enclosing scope, the compiler
raises a SyntaxError. (It may be a bit of a stretch to call this a
syntax error, but so far SyntaxError is used for all compile-time
errors, including, for example, __future__ import with an unknown
feature name.) Guido has said that this kind of declaration in the
absence of an outer binding should be considered an error[38].

If a nonlocal declaration collides with the name of a formal parameter
in the local scope, the compiler raises a SyntaxError.

A shorthand form is also permitted, in which nonlocal is prepended to an
assignment or augmented assignment:

    nonlocal x = 3

The above has exactly the same meaning as nonlocal x; x = 3. (Guido
supports a similar form of the global statement[39].)

On the left side of the shorthand form, only identifiers are allowed,
not target expressions like x[0]. Otherwise, all forms of assignment are
allowed. The proposed grammar of the nonlocal statement is:

    nonlocal_stmt ::=
        "nonlocal" identifier ("," identifier)*
                   ["=" (target_list "=")+ expression_list]
      | "nonlocal" identifier augop expression_list

The rationale for allowing all these forms of assignment is that it
simplifies understanding of the nonlocal statement. Separating the
shorthand form into a declaration and an assignment is sufficient to
understand what it means and whether it is valid.

Note

The shorthand syntax was not added in the original implementation of the
PEP. Later discussions[40][41] concluded this syntax should not be
implemented.

Backward Compatibility

This PEP targets Python 3000, as suggested by Guido[42]. However, others
have noted that some options considered in this PEP may be small enough
changes to be feasible in Python 2.x[43], in which case this PEP could
possibly be moved to be a 2.x series PEP.

As a (very rough) measure of the impact of introducing a new keyword,
here is the number of times that some of the proposed keywords appear as
identifiers in the standard library, according to a scan of the Python
SVN repository on November 5, 2006:

    nonlocal    0
    use         2
    using       3
    reuse       4
    free        8
    outer     147

global appears 214 times as an existing keyword. As a measure of the
impact of using global as the outer-scope keyword, there are 18 files in
the standard library that would break as a result of such a change
(because a function declares a variable global before that variable has
been introduced in the global scope):

    cgi.py
    dummy_thread.py
    mhlib.py
    mimetypes.py
    idlelib/PyShell.py
    idlelib/run.py
    msilib/__init__.py
    test/inspect_fodder.py
    test/test_compiler.py
    test/test_decimal.py
    test/test_descr.py
    test/test_dummy_threading.py
    test/test_fileinput.py
    test/test_global.py (not counted: this tests the keyword itself)
    test/test_grammar.py (not counted: this tests the keyword itself)
    test/test_itertools.py
    test/test_multifile.py
    test/test_scope.py (not counted: this tests the keyword itself)
    test/test_threaded_import.py
    test/test_threadsignals.py
    test/test_warnings.py

References

[15] Explicit Lexical Scoping (pre-PEP?) (Guido van Rossum)
https://mail.python.org/pipermail/python-dev/2006-July/066995.html

Acknowledgements

The ideas and proposals mentioned in this PEP are gleaned from countless
Python-Dev postings. Thanks to Jim Jewett, Mike Orr, Jason Orendorff,
and Christian Tanzer for suggesting specific edits to this PEP.

Copyright

This document has been placed in the public domain.

[1] Scoping (was Re: Lambda binding solved?) (Rafael Bracho)
https://legacy.python.org/search/hypermail/python-1994q1/0301.html

[2] Extended Function syntax (Just van Rossum)
https://mail.python.org/pipermail/python-dev/2003-February/032764.html

[3] Lexical scoping in Python 3k (Guido van Rossum)
https://mail.python.org/pipermail/python-dev/2006-July/066950.html

[4] Ruby 2.0 block local variable
https://web.archive.org/web/20070105131417/http://redhanded.hobix.com/inspect/ruby20BlockLocalVariable.html

[5] Better Control of Nested Lexical Scopes (Almann T. Goo)
https://mail.python.org/pipermail/python-dev/2006-February/061568.html

[6] Better Control of Nested Lexical Scopes (Almann T. Goo)
https://mail.python.org/pipermail/python-dev/2006-February/061568.html

[7] Lexical scoping in Python 3k (Ka-Ping Yee)
https://mail.python.org/pipermail/python-dev/2006-July/066862.html

[8] Explicit Lexical Scoping (pre-PEP?) (Talin)
https://mail.python.org/pipermail/python-dev/2006-July/066978.html

[9] Sky pie: a "var" keyword (a thread started by Neil Toronto)
https://mail.python.org/pipermail/python-3000/2006-October/003968.html

[10] Using and binding relative names (Phillip J. Eby)
https://mail.python.org/pipermail/python-dev/2006-February/061636.html

[11] Using and binding relative names (Almann T. Goo)
https://mail.python.org/pipermail/python-dev/2006-February/061761.html

[12] Using and binding relative names (Steven Bethard)
https://mail.python.org/pipermail/python-dev/2006-February/061749.html

[13] Explicit Lexical Scoping (pre-PEP?) (Guido van Rossum)
https://mail.python.org/pipermail/python-dev/2006-July/067004.html

[14] Extended Function syntax (Just van Rossum)
https://mail.python.org/pipermail/python-dev/2003-February/032764.html

[15] Scoping (was Re: Lambda binding solved?) (Rafael Bracho)
https://legacy.python.org/search/hypermail/python-1994q1/0301.html

[16] Closure semantics (Guido van Rossum)
https://mail.python.org/pipermail/python-dev/2003-October/039214.html

[17] PEP for Better Control of Nested Lexical Scopes (Jeremy Hylton)
https://mail.python.org/pipermail/python-dev/2006-February/061602.html

[18] PEP for Better Control of Nested Lexical Scopes (Almann T. Goo)
https://mail.python.org/pipermail/python-dev/2006-February/061603.html

[19] Lexical scoping in Python 3k (Ka-Ping Yee)
https://mail.python.org/pipermail/python-dev/2006-July/066862.html

[20] Lexical scoping in Python 3k (Greg Ewing)
https://mail.python.org/pipermail/python-dev/2006-July/066889.html

[21] Lexical scoping in Python 3k (Ka-Ping Yee)
https://mail.python.org/pipermail/python-dev/2006-July/066942.html

[22] Explicit Lexical Scoping (pre-PEP?) (Andrew Clover)
https://mail.python.org/pipermail/python-dev/2006-July/067007.html

[23] Explicit Lexical Scoping (pre-PEP?) (Andrew Clover)
https://mail.python.org/pipermail/python-dev/2006-July/067007.html

[24] Explicit Lexical Scoping (pre-PEP?) (Andrew Clover)
https://mail.python.org/pipermail/python-dev/2006-July/067007.html

[25] Explicit Lexical Scoping (pre-PEP?) (Matthew Barnes)
https://mail.python.org/pipermail/python-dev/2006-July/067221.html

[26] Alternatives to 'outer' (Talin)
https://mail.python.org/pipermail/python-3000/2006-October/004021.html

[27] Alternatives to 'outer' (Talin)
https://mail.python.org/pipermail/python-3000/2006-October/004021.html

[28] Alternatives to 'outer' (Talin)
https://mail.python.org/pipermail/python-3000/2006-October/004021.html

[29] Alternatives to 'outer' (Talin)
https://mail.python.org/pipermail/python-3000/2006-October/004021.html

[30] Alternatives to 'outer' (Talin)
https://mail.python.org/pipermail/python-3000/2006-October/004021.html

[31] Alternatives to 'outer' (Talin)
https://mail.python.org/pipermail/python-3000/2006-October/004021.html

[32] Alternatives to 'outer' (Talin)
https://mail.python.org/pipermail/python-3000/2006-October/004021.html

[33] Alternatives to 'outer' (Jim Jewett)
https://mail.python.org/pipermail/python-3000/2006-November/004153.html

[34] Draft PEP for outer scopes (Talin)
https://mail.python.org/pipermail/python-3000/2006-November/004190.html

[35] Global variable (version 2006-11-01T01:23:16)
https://en.wikipedia.org/w/index.php?title=Global_variable&oldid=85001451

[36] Explicit Lexical Scoping (pre-PEP?) (Guido van Rossum)
https://mail.python.org/pipermail/python-dev/2006-July/066991.html

[37] Explicit Lexical Scoping (pre-PEP?) (Guido van Rossum)
https://mail.python.org/pipermail/python-dev/2006-July/067067.html

[38] Lexical scoping in Python 3k (Guido van Rossum)
https://mail.python.org/pipermail/python-dev/2006-July/066968.html

[39] Draft PEP for outer scopes (Guido van Rossum)
https://mail.python.org/pipermail/python-3000/2006-November/004166.html

[40] Issue 4199: combining assignment with global & nonlocal (Guido van
Rossum)
https://mail.python.org/pipermail/python-dev/2013-June/127142.html

[41] Whatever happened to 'nonlocal x = y'? (Guido van Rossum)
https://mail.python.org/pipermail/python-dev/2018-January/151627.html

[42] Explicit Lexical Scoping (pre-PEP?) (Guido van Rossum)
https://mail.python.org/pipermail/python-dev/2006-July/067067.html

[43] Draft PEP for outer scopes (Alyssa Coghlan)
https://mail.python.org/pipermail/python-3000/2006-November/004237.html