PEP: 762 Title: REPL-acing the default REPL Author: Pablo Galindo
Salgado <pablogsal@python.org>, Łukasz Langa <lukasz@python.org>,
Lysandros Nikolaou <lisandrosnik@gmail.com>, Emily Morehouse-Valcarcel
<emily@python.org> Sponsor: Pablo Galindo Salgado Status: Final Type:
Informational Created: 11-Oct-2024 Python-Version: 3.13

Abstract

One of Python’s core strengths is its interactive mode, also known as
the Read-Eval-Print Loop (REPL), or the Python console, or the Python
shell. This PEP describes the new implementation of this functionality
written in Python. The new REPL released in Python 3.13 aims to provide
modern features expected by today's users, such as multi-line editing,
syntax highlighting, custom commands, and an overall improved
interactive experience.

Motivation

Up to Python 3.12, the interactive shell of CPython was written in C as
a special mode of the parser. It was therefore difficult to maintain and
extend. It relied on the existence of GNU readline (or an equivalent)
for basic functionality like cursor movement and history tracking.
Python compiled without this library provided an interactive mode of
very limited capability. On the other hand, Python compiled with
readline outsourced decisions and configuration around user input in
ways that made extending it difficult.

This complexity has deterred contributions and has made it challenging
to implement new features. As a result, the CPython interactive shell
has seen minimal changes, falling behind user expectations for modern
equivalents.

Many features that users have come to expect from modern REPLs were
absent in the previous version. Some examples of these features include
multi-line editing and history, custom commands, syntax highlighting, or
ergonomic handling of copy and paste. The lack of these features greatly
impacts the user experience of many groups of users of CPython, in
particular in environments where users don’t control dependencies and
cannot install their own packages. This is especially common for users
learning the language and educators.

Addressing such issues with the C implementation would require complex
workarounds, such as AST matching of commands, which would add
prohibitive complexity to the codebase.

With the new REPL written in Python, we are addressing these limitations
while also bringing CPython's interactive experience more in line with
modern expectations and capabilities.

Rationale

Implementing the new REPL in Python, rather than C, has significantly
lowered the barrier to entry for contributors. This change has made it
easier to test, validate, and modify the REPL, leading to increased
community involvement and faster feature development. The improved
accessibility of the codebase is expected to result in a more rapidly
evolving and user-responsive REPL.

Instead of writing a Python REPL from scratch, we decided to base the
implementation of the new REPL on PyREPL. This decision was driven by
several key factors. First and foremost, developing a terminal
application that works consistently across different operating systems
and terminal emulators is a complex undertaking. By adopting PyREPL,
which has been battle-tested in the PyPy project, we can leverage
existing, proven code rather than starting from scratch.

Sharing a codebase with PyPy for the REPL implementation offers mutual
benefits to both projects. It allows for shared maintenance efforts,
faster bug fixes, and feature improvements that can benefit users of
both CPython and PyPy. This collaboration can lead to a more robust and
feature-rich REPL for the entire Python ecosystem.

The previous REPL written in C leveraged the “readline” or “editline”
libraries as a backend to allow certain features such as navigation,
history preservation and recall, autocompletion, and configurable
keyboard behavior. PyREPL does not use those libraries, implementing
most of the other functionality directly as part of the shell. The main
missing functionality (configurability of input) is outweighed by the
benefits of the new architecture. The configuration files for these
libraries (e.g. inputrc) are complex and include features that PyREPL
doesn’t plan to implement, making it infeasible to transparently add
support for them in the new shell. Using “readline” or “editline” in
PyREPL would be prohibitively complex due to multi-line editing handling
and multiplatform support.

Although this means that existing readline/editline configurations will
not be compatible with the PyREPL, we believe the enhanced features and
improved extensibility are an overall win. See “Backwards Compatibility”
for discussion of continued support for customized workflows.

The previous REPL made it challenging to properly implement custom
commands, which is a very common feature of interactive shells. For
instance, the exit command was implemented as a method call of a custom
object injected in the global namespace, leading to unintuitive behavior
that often confuses users when they simply type exit, as the interpreter
prompts them to the supposedly correct usage exit().

Specification

PyREPL is implemented as a new private Python module called !_pyrepl,
existing alongside the current C implementation. In its first
implementation, it introduces the following key features:

1.  Multi-line History and Editing: Users are able to navigate and edit
    their command history across multiple lines, improving the ability
    to refine and reuse complex blocks of code.

    Editing multi-line blocks provides automatic indentation using four
    spaces, which is consistent with PEP 8 recommendations. When a line
    ending with a colon is encountered, the following line is
    automatically indented utilizing the indentation pattern that is
    inferred from the first line that contains indentation. Lines are
    indented with four spaces, and tabs are converted into spaces.

    Users can access history of commands by using up and down arrows.
    Within a multi-line entry, the arrow keys navigate line-by-line
    within the block before moving to the next history entry. The down
    arrow works in reverse, navigating from older entries to the most
    recent.

    History can be searched forward (using Ctrl+S) and in reverse (using
    Ctrl+R) using a custom-specified substring query. It can also be
    searched with a prefix query by entering the prefix into a shell
    line and using PageUp and PageDown keys.

2.  Copying and Pasting: in supported terminal emulators, bracketed
    pasting capability is discovered and used by PyREPL. This allows for
    transparent pasting of blocks of code without immediate execution or
    invalid automatic indentation.

    For terminal emulators that don’t support this mode, a dedicated
    paste mode is implemented to allow for easy insertion of multi-line
    code snippets without triggering immediate execution or indentation
    issues.

    Users enter manual paste mode by hitting the F3 key. The prompt
    changes from >>> to (paste) where users can paste contents from
    their clipboard or manually type as desired. Once the content is
    ready, hitting F3 exits paste mode. Then, pressing Enter executes
    the block.

    Users can enter multiple commands on a single input when using paste
    mode, which will help paste code from other sources.

    To copy blocks of code without the leading command prompts and
    without the output of the commands, users can enter the history view
    via the F2 key. This mode uses a pager to display history of
    executed commands without the prompts and output.

3.  Help via F1.

    Access to the standard Help module is accessible via a Custom
    Command help (see below) or via the F1 key. Hit F1 to enter help
    mode. When you're done, hit F1 or a standard command (q, quit or
    exit) to exit.

    Browsing interactive help does not retain command history.

4.  Custom Commands: The REPL supports the implementation of custom
    commands, such as exit, in a more natural and user-friendly manner,
    avoiding the current function call workaround.

    The initial list of custom commands includes:

    -   exit
    -   quit
    -   copyright
    -   help
    -   clear

    Commands are available as long as there is no name conflict with a
    variable in a reachable scope. For example, after assigning
    exit = 1, the variable will take precedence over PyREPL commands.
    del exit in this case will remove the conflict and the command will
    function again.

5.  Colors: the prompts as well as certain elements of the output, like
    exception tracebacks, are now colored. Colors can be disabled using
    the standard NO_COLOR environment variable, or forced by using the
    standard FORCE_COLOR environment variable. A Python-specific
    environment variable is also available called PYTHON_COLORS. The
    initial implementation in Python 3.13 does not offer customization
    of the color theme.

These features are significantly enhancing the interactive Python
experience, bringing it more in line with modern development
environments and user expectations. The implementation is in Python,
offering several advantages:

1.  Easier Testing and Validation: Writing tests for Python code is
    dramatically simpler and more straightforward than for C code,
    allowing for more comprehensive test coverage of all existing and
    old features.
2.  Lower Contribution Barrier: Python's accessibility compared to C has
    been encouraging more community contributions, leading to faster
    feature development and bug fixes.
3.  Flexibility: A Python implementation is easier to extend and modify,
    improving developer velocity on new features and improvements by
    core developers and contributors alike.

Backwards Compatibility

The PyREPL implementation is designed to maintain full backward
compatibility with existing Python code as the old basic REPL will be
preserved as a fallback and is available on demand, in case custom
workflows require it. It will also be used in cases where the new REPL
cannot be used due to environmental constraints or other issues. Users
have the option to explicitly choose the old basic REPL by setting the
environment variable PYTHON_BASIC_REPL to 1. This ensures that users can
continue using the familiar interface and capabilities if they prefer,
or if they encounter any issues with the new implementation.

It's important to emphasize that the introduction of PyREPL does not
remove any existing functionality. Any functionality of the old basic
REPL unavailable in PyREPL is preserved and maintained in the old basic
REPL that can be used by users as a fallback.

In particular, users wanting to continue using their custom input
configuration in inputrc or editrc files can continue using the old
basic REPL.

The authors do not expect any PyREPL functionality to be ported to the
old basic REPL. Similarly, inputrc and editrc support is explicitly not
planned in PyREPL. Those configuration files are provided by and parsed
by “readline” and “editline” libraries, and their functional scope does
not match the functionality PyREPL is targeting.

To facilitate a smooth transition, clear documentation is provided on
how to switch between PyREPL and the old basic REPL.

This approach ensures that while we're introducing significant
improvements with the new REPL, we're not forcing any immediate changes
on users who rely on the current implementation. The fallback mechanism
and user choice option provide a safety net that allows for gradual
adoption of the new REPL while maintaining all existing functionality.

Security Implications

There are no security implications derived from this proposal.

How to Teach This

The introduction of PyREPL is accompanied by documentation and
tutorials. Key areas of focus for education will include:

1.  Detailed explanations on using multi-line editing, paste mode, and
    other new features.
2.  Custom commands (existing and new).
3.  How to switch to the new REPL, including any differences from the
    previous readline/editline-based configuration.

Rejected Ideas

Several alternative approaches were considered and ultimately rejected:

1.  Extending the current C implementation: While this would maintain
    maximum backwards compatibility, it was deemed too complex and would
    not address the fundamental limitations described ut supra.
2.  Developing a new REPL from scratch: This approach was rejected due
    to the complexity of creating a cross-platform terminal application
    and the desire to leverage existing, proven code.
3.  Using other existing REPL implementations: The authors looked at
    several alternatives like IPython, bpython, ptpython, and xonsh.
    While all the above are impressive projects, in the end PyREPL was
    chosen for its combination of maturity, feature set, and lack of
    additional dependencies. Another key factor was the alignment with
    PyPy's implementation.

Acknowledgments

Thanks to Diego Russo for providing feedback on drafts of this PEP.

Copyright

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.