PEP: 667 Title: Consistent views of namespaces Author: Mark Shannon
<mark@hotpy.org>, Tian Gao <gaogaotiantian@hotmail.com> Discussions-To:
https://discuss.python.org/t/46631 Status: Accepted Type: Standards
Track Created: 30-Jul-2021 Python-Version: 3.13 Post-History:
20-Aug-2021 Resolution:
https://discuss.python.org/t/pep-667-consistent-views-of-namespaces/46631/25

Abstract

In early versions of Python all namespaces, whether in functions,
classes or modules, were all implemented the same way: as a dictionary.

For performance reasons, the implementation of function namespaces was
changed. Unfortunately this meant that accessing these namespaces
through locals() and frame.f_locals ceased to be consistent and some odd
bugs crept in over the years as threads, generators and coroutines were
added.

This PEP proposes making these namespaces consistent once more.
Modifications to frame.f_locals will always be visible in the underlying
variables. Modifications to local variables will immediately be visible
in frame.f_locals, and they will be consistent regardless of threading
or coroutines.

The locals() function will act the same as it does now for class and
modules scopes. For function scopes it will return an instantaneous
snapshot of the underlying frame.f_locals.

Motivation

The current implementation of locals() and frame.f_locals is slow,
inconsistent and buggy. We want to make it faster, consistent, and most
importantly fix the bugs.

For example:

    class C:
        x = 1
        sys._getframe().f_locals['x'] = 2
        print(x)

prints 2

but:

    def f():
        x = 1
        sys._getframe().f_locals['x'] = 2
        print(x)
    f()

prints 1

This is inconsistent, and confusing. With this PEP both examples would
print 2.

Worse than that, the current behavior can result in strange bugs.

There are no compensating advantages for the current behavior; it is
unreliable and slow.

Rationale

The current implementation of frame.f_locals returns a dictionary that
is created on the fly from the array of local variables. This can result
in the array and dictionary getting out of sync with each other. Writes
to the f_locals may not show up as modifications to local variables.
Writes to local variables can get lost.

By making frame.f_locals return a view on the underlying frame, these
problems go away. frame.f_locals is always in sync with the frame
because it is a view of it, not a copy of it.

Specification

Python

frame.f_locals will return a view object on the frame that implements
the collections.abc.Mapping interface.

For module and class scopes frame.f_locals will be a dictionary, for
function scopes it will be a custom class.

locals() will be defined as:

    def locals():
        frame = sys._getframe(1)
        f_locals = frame.f_locals
        if frame.is_function():
            f_locals = dict(f_locals)
        return f_locals

All writes to the f_locals mapping will be immediately visible in the
underlying variables. All changes to the underlying variables will be
immediately visible in the mapping. The f_locals object will be a full
mapping, and can have arbitrary key-value pairs added to it.

For example:

    def l():
        "Get the locals of caller"
        return sys._getframe(1).f_locals

    def test():
        if 0: y = 1 # Make 'y' a local variable
        x = 1
        l()['x'] = 2
        l()['y'] = 4
        l()['z'] = 5
        y
        print(locals(), x)

test() will print {'x': 2, 'y': 4, 'z': 5} 2.

In Python 3.10, the above will fail with an UnboundLocalError, as the
definition of y by l()['y'] = 4 is lost.

If the second-to-last line were changed from y to z, this would be a
NameError, as it is today. Keys added to frame.f_locals that are not
lexically local variables remain visible in frame.f_locals, but do not
dynamically become local variables.

C-API

Extensions to the API

Three new C-API functions will be added:

    PyObject *PyEval_GetFrameLocals(void)
    PyObject *PyEval_GetFrameGlobals(void)
    PyObject *PyEval_GetFrameBuiltins(void)

PyEval_GetFrameLocals() is equivalent to: locals().
PyEval_GetFrameGlobals() is equivalent to: globals().

All these functions will return a new reference.

Changes to existing APIs

PyFrame_GetLocals(f) is equivalent to f.f_locals, and hence its return
value will change as described above for accessing f.f_locals.

The following C-API functions will be deprecated, as they return
borrowed references:

    PyEval_GetLocals()
    PyEval_GetGlobals()
    PyEval_GetBuiltins()

The following functions should be used instead:

    PyEval_GetFrameLocals()
    PyEval_GetFrameGlobals()
    PyEval_GetFrameBuiltins()

which return new references.

The semantics of PyEval_GetLocals() is changed as it now returns a proxy
for the frame locals in optimized frames, not a dictionary.

The following three functions will become no-ops, and will be
deprecated:

    PyFrame_FastToLocalsWithError()
    PyFrame_FastToLocals()
    PyFrame_LocalsToFast()

Behavior of f_locals for optimized functions

Although f.f_locals behaves as if it were the namespace of the function,
there will be some observable differences. For example,
f.f_locals is f.f_locals may be False.

However f.f_locals == f.f_locals will be True, and all changes to the
underlying variables, by any means, will always be visible.

Backwards Compatibility

Python

The current implementation has many corner cases and oddities. Code that
works around those may need to be changed. Code that uses locals() for
simple templating, or print debugging, will continue to work correctly.
Debuggers and other tools that use f_locals to modify local variables,
will now work correctly, even in the presence of threaded code,
coroutines and generators.

C-API

PyEval_GetLocals

Because PyEval_GetLocals() returns a borrowed reference, it requires the
proxy mapping to be cached on the frame, extending its lifetime and
creating a cycle. PyEval_GetFrameLocals() should be used instead.

This code:

    locals = PyEval_GetLocals();
    if (locals == NULL) {
        goto error_handler;
    }
    Py_INCREF(locals);

should be replaced with:

    locals = PyEval_GetFrameLocals();
    if (locals == NULL) {
        goto error_handler;
    }

Implementation

Each read of frame.f_locals will create a new proxy object that gives
the appearance of being the mapping of local (including cell and free)
variable names to the values of those local variables.

A possible implementation is sketched out below. All attributes that
start with an underscore are invisible and cannot be accessed directly.
They serve only to illustrate the proposed design.

    NULL: Object # NULL is a singleton representing the absence of a value.

    class CodeType:

        _name_to_offset_mapping_impl: dict | NULL
        _cells: frozenset # Set of indexes of cell and free variables
        ...

        def __init__(self, ...):
            self._name_to_offset_mapping_impl = NULL
            self._variable_names = deduplicate(
                self.co_varnames + self.co_cellvars + self.co_freevars
            )
            ...

        @property
        def _name_to_offset_mapping(self):
            "Mapping of names to offsets in local variable array."
            if self._name_to_offset_mapping_impl is NULL:
                self._name_to_offset_mapping_impl = {
                    name: index for (index, name) in enumerate(self._variable_names)
                }
            return self._name_to_offset_mapping_impl

    class FrameType:

        _locals : array[Object] # The values of the local variables, items may be NULL.
        _extra_locals: dict | NULL # Dictionary for storing extra locals not in _locals.
        _locals_cache: FrameLocalsProxy | NULL # required to support PyEval_GetLocals()

        def __init__(self, ...):
            self._extra_locals = NULL
            self._locals_cache = NULL
            ...

        @property
        def f_locals(self):
            return FrameLocalsProxy(self)

    class FrameLocalsProxy:
        "Implements collections.MutableMapping."

        __slots__ = ("_frame", )

        def __init__(self, frame:FrameType):
            self._frame = frame

        def __getitem__(self, name):
            f = self._frame
            co = f.f_code
            if name in co._name_to_offset_mapping:
                index = co._name_to_offset_mapping[name]
                val = f._locals[index]
                if val is NULL:
                    raise KeyError(name)
                if index in co._cells
                    val = val.cell_contents
                    if val is NULL:
                        raise KeyError(name)
                return val
            else:
                if f._extra_locals is NULL:
                    raise KeyError(name)
                return f._extra_locals[name]

        def __setitem__(self, name, value):
            f = self._frame
            co = f.f_code
            if name in co._name_to_offset_mapping:
                index = co._name_to_offset_mapping[name]
                kind = co._local_kinds[index]
                if index in co._cells
                    cell = f._locals[index]
                    cell.cell_contents = val
                else:
                    f._locals[index] = val
            else:
                if f._extra_locals is NULL:
                    f._extra_locals = {}
                f._extra_locals[name] = val

        def __iter__(self):
            f = self._frame
            co = f.f_code
            yield from iter(f._extra_locals)
            for index, name in enumerate(co._variable_names):
                val = f._locals[index]
                if val is NULL:
                    continue
                if index in co._cells:
                    val = val.cell_contents
                    if val is NULL:
                        continue
                yield name

        def __contains__(self, item):
            f = self._frame
            if item in f._extra_locals:
                return True
            return item in co._variable_names

        def __len__(self):
            f = self._frame
            co = f.f_code
            res = 0
            for index, _ in enumerate(co._variable_names):
                val = f._locals[index]
                if val is NULL:
                    continue
                if index in co._cells:
                    if val.cell_contents is NULL:
                        continue
                res += 1
            return len(self._extra_locals) + res

C API

PyEval_GetLocals() will be implemented roughly as follows:

    PyObject *PyEval_GetLocals(void) {
        PyFrameObject * = ...; // Get the current frame.
        if (frame->_locals_cache == NULL) {
            frame->_locals_cache = PyEval_GetFrameLocals();
        }
        return frame->_locals_cache;
    }

As with all functions that return a borrowed reference, care must be
taken to ensure that the reference is not used beyond the lifetime of
the object.

Impact on PEP 709 inlined comprehensions

For inlined comprehensions within a function, locals() currently behaves
the same inside or outside of the comprehension, and this will not
change. The behavior of locals() inside functions will generally change
as specified in the rest of this PEP.

For inlined comprehensions at module or class scope, currently calling
locals() within the inlined comprehension returns a new dictionary for
each call. This PEP will make locals() within a function also always
return a new dictionary for each call, improving consistency; class or
module scope inlined comprehensions will appear to behave as if the
inlined comprehension is still a distinct function.

Comparison with PEP 558

This PEP and PEP 558 share a common goal: to make the semantics of
locals() and frame.f_locals() intelligible, and their operation
reliable.

The key difference between this PEP and PEP 558 is that PEP 558 keeps an
internal copy of the local variables, whereas this PEP does not.

PEP 558 does not specify exactly when the internal copy is updated,
making the behavior of PEP 558 impossible to reason about.

Implementation

The implementation is in development as a draft pull request on GitHub.

Copyright

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.