PEP: 660 Title: Editable installs for pyproject.toml based builds (wheel
based) Author: Daniel Holth <dholth@gmail.com>, Stéphane Bidoul
<stephane.bidoul@gmail.com> Sponsor: Paul Moore <p.f.moore@gmail.com>
Discussions-To:
https://discuss.python.org/t/draft-pep-editable-installs-for-pep-517-style-build-backends/8510
Status: Final Type: Standards Track Topic: Packaging Content-Type:
text/x-rst Created: 30-Mar-2021 Post-History: Resolution:
https://discuss.python.org/t/pronouncement-on-peps-660-and-662-editable-installs/9450

Abstract

This document describes a PEP 517 style method for the installation of
packages in editable mode.

Motivation

Python programmers want to be able to develop packages without having to
install (i.e. copy) them into site-packages, for example, by working in
a checkout of the source repository.

While this can be done by adding the relevant source directories to
PYTHONPATH, setuptools provides the setup.py develop mechanism that
makes the process easier, and also installs dependencies and entry
points such as console scripts. pip exposes this mechanism via its
pip install --editable option.

The installation of projects in such a way that the python code being
imported remains in the source directory is known as the editable
installation mode.

Now that PEP 517 provides a mechanism to create alternatives to
setuptools, and decouple installation front ends from build backends, we
need a new mechanism to install packages in editable mode.

Rationale

PEP 517 deferred "Editable installs", meaning non-setup.py distributions
lacked that feature. The only way to retain editable installs for these
distributions was to provide a compatible setup.py develop
implementation. By defining an editable hook other build frontends gain
parity with setup.py.

Terminology and goals

The editable installation mode implies that the source code of the
project being installed is available in a local directory.

Once the project is installed in editable mode, users expect that
changes to the project python code in the local source tree become
effective without the need of a new installation step.

Some kind of changes, such as the addition or modification of entry
points, or the addition of new dependencies, require a new installation
step to become effective. These changes are typically made in build
backend configuration files (such as pyproject.toml), so it is
consistent with the general user expectation that python source code is
imported from the source tree.

The modification of non-python source code such a C extension modules
obviously require a compilation and/or installation step to become
effective. The exact steps to perform will remain specific to the build
backend used.

When a project is installed in editable mode, users expect the
installation to behave identically as a regular installation. In
particular the code must be importable by other code, and metadata must
be available to standard mechanisms such as importlib.metadata.

Depending on the way build backends implement this specification, some
minor differences may be visible such as the presence of additional
files that are in the source tree and would not be part of a regular
install. Build backends are encouraged to document such potential
differences.

The Mechanism

This PEP adds three optional hooks to the PEP 517 backend interface.
These hooks are used to build a wheel that, when installed, allows that
distribution to be imported from its source folder.

build_editable

    def build_editable(wheel_directory, config_settings=None, metadata_directory=None):
        ...

Must build a .whl file, and place it in the specified wheel_directory.
It must return the basename (not the full path) of the .whl file it
creates, as a unicode string.

May do an in-place build of the distribution as a side effect so that
any extension modules or other built artifacts are ready to be used.

The .whl file must comply with the Wheel binary file format
specification (PEP 427). In particular it must contain a compliant
.dist-info directory. Metadata must be identical as the one that would
have been produced by build_wheel or prepare_metadata_for_build_wheel,
except for Requires-Dist which may differ slightly as explained below.

Build-backends must produce wheels that have the same dependencies
(Requires-Dist metadata) as wheels produced by the build_wheel hook,
with the exception that they can add dependencies necessary for their
editable mechanism to function at runtime (such as editables).

The filename for the "editable" wheel needs to be PEP 427 compliant too.
It does not need to use the same tags as build_wheel but it must be
tagged as compatible with the system.

If the build frontend has previously called
prepare_metadata_for_build_editable and depends on the wheel resulting
from this call to have metadata matching this earlier call, then it
should provide the path to the created .dist-info directory as the
metadata_directory argument. If this argument is provided, then
build_editable MUST produce a wheel with identical metadata. The
directory passed in by the build frontend MUST be identical to the
directory created by prepare_metadata_for_build_editable, including any
unrecognized files it created.

An "editable" wheel uses the wheel format not for distribution but as
ephemeral communication between the build system and the front end. This
avoids having the build backend install anything directly. This wheel
must not be exposed to end users, nor cached, nor distributed.

get_requires_for_build_editable

    def get_requires_for_build_editable(config_settings=None):
        ...

This hook MUST return an additional list of strings containing PEP 508
dependency specifications, above and beyond those specified in the
pyproject.toml file, to be installed when calling the build_editable
hooks.

If not defined, the default implementation is equivalent to return [].

prepare_metadata_for_build_editable

    def prepare_metadata_for_build_editable(metadata_directory, config_settings=None):
        ...

Must create a .dist-info directory containing wheel metadata inside the
specified metadata_directory (i.e., creates a directory like
{metadata_directory}/{package}-{version}.dist-info/). This directory
MUST be a valid .dist-info directory as defined in the wheel
specification, except that it need not contain RECORD or signatures. The
hook MAY also create other files inside this directory, and a build
frontend MUST preserve, but otherwise ignore, such files; the intention
here is that in cases where the metadata depends on build-time
decisions, the build backend may need to record these decisions in some
convenient format for re-use by the actual wheel-building step.

This must return the basename (not the full path) of the .dist-info
directory it creates, as a unicode string.

If a build frontend needs this information and the method is not
defined, it should call build_editable and look at the resulting
metadata directly.

What to put in the wheel

Build backends must populate the generated wheel with files that when
installed will result in an editable install. Build backends may use
different techniques to achieve the goals of an editable install. This
section provides examples and is not normative.

-   Build backends may choose to place a .pth file at the root of the
    .whl file, containing the root directory of the source tree. This
    approach is simple but not very precise, although it may be
    considered good enough (especially when using the src layout) and is
    similar to what setup.py develop currently does.
-   The editables library shows how to build proxy modules that provide
    a high quality editable installation. It accepts a list of modules
    to include, and hide. When imported, these proxy modules replace
    themselves with the code from the source tree. Path-based methods
    make all scripts under a path importable, often including the
    project's own setup.py and other scripts that would not be part of a
    normal installation. The proxy strategy can achieve a higher level
    of fidelity than path-based methods.
-   Symbolic links are another useful mechanism to realize editable
    installs. Since, at the time this writing, the wheel specification
    does not support symbolic links, they are not directly usable to
    set-up symbolic links in the target environment. It is however
    possible for the backend to create a symlink structure in some build
    directory of the source tree, and add that directory to the python
    path via a .pth file in the "editable" wheel. If some files linked
    in this manner depend on python implementation or version, ABI or
    platform, care must be taken to generate the link structure in
    different directories depending on compatibility tags, so the same
    project tree can be installed in editable mode in multiple
    environments.

Frontend requirements

Frontends must install "editable" wheels in the same way as regular
wheels. This also means uninstallation of editables does not require any
special treatment.

Frontends must create a direct_url.json file in the .dist-info directory
of the installed distribution, in compliance with PEP 610. The url value
must be a file:// url pointing to the project directory (i.e. the
directory containing pyproject.toml), and the dir_info value must be
{'editable': true}.

Frontends must execute get_requires_for_build_editable hooks in an
environment which contains the bootstrap requirements specified in the
pyproject.toml file.

Frontends must execute the prepare_metadata_for_build_editable and
build_editable hooks in an environment which contains the bootstrap
requirements from pyproject.toml and those specified by the
get_requires_for_build_editable hook.

Frontends must not expose the wheel obtained from build_editable to end
users. The wheel must be discarded after installation and must not be
cached nor distributed.

Limitations

With regard to the wheel .data directory, this PEP focuses on making the
purelib and platlib categories (installed into site-packages)
"editable". It does not make special provision for the other categories
such as headers, data and scripts. Package authors are encouraged to use
console_scripts, make their scripts tiny wrappers around library
functionality, or manage these from the source checkout during
development.

Prototypes

At the time of writing this PEP, several prototype implementations are
available in various frontends and backends. We provide links below to
illustrate possible approaches.

Frontends:

-   pip (pull request)

Build backends:

-   enscons (pull request 1, pull request 2)
-   flit (pull request)
-   hatchling (sdist)
-   pdm (pull request)
-   setuptools (setuptools_pep660 repository)

Rejected ideas

editable local version identifier

The ideas of having build backends append or modify the local version
identifier to include the editable string has been rejected because it
would not satisfy == version speicifier that include the local version
identifier. In other words pkg==1.0+local is not satisfied by version
1.0+local.editable.

Virtual wheel

Another approach was proposed in PEP 662, where the build backend
returns a mapping from source files and directories to the installed
layout. It is then up to the installer frontend to realize the editable
installation by whatever means it deems adequate for its users.

In terms of capabilities, both proposals provide the core "editable"
feature.

The key difference is that PEP 662 leaves it to the frontend to decide
how the editable installation will be realized, while with this PEP, the
choice must be made by the backend. Both approaches can in principle
provide several editable installation methods for a given project, and
let the developer choose one at install time.

At the time of writing this PEP, it is clear that the community has a
wide range of theoretical and practical expectations about editable
installs. The reality is that the only one there is wide experience with
is path insertion via .pth (i.e. what setup.py develop does).

We believe that PEP 660 better addresses these "unknown unknowns" today
in the most reliable way, by letting project authors select the backend
or implement the method that provides the editable mechanism that best
suit their requirements, and test it works correctly. Since the frontend
has no latitude in how to install the "editable" wheel, in case of
issue, there is only one place to investigate: the build backend.

With PEP 662, issues need to be investigated in the frontend, the
backend and possiblty the specification. There is also a high
probability that different frontends, implementing the specification in
different ways, will produce installations that behave differently than
project authors intended, creating confusion, or worse, projects that
only work with specific frontends or IDEs.

Unpacked wheel

A prototype was made that created an unpacked wheel in a temporary
directory, to be copied to the target environment by the frontend. This
approach was not pursued because a wheel archive is easy to create for
the backend, and using a wheel as communication mechanism is a better
fit with the PEP 517 philosophy, and therefore keeps things simpler for
the frontend.

References

Copyright

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.



  Local Variables: mode: indented-text indent-tabs-mode: nil
  sentence-end-double-space: t fill-column: 70 coding: utf-8 End: