PEP: 650 Title: Specifying Installer Requirements for Python Projects
Author: Vikram Jayanthi <vikramjayanthi@google.com>, Dustin Ingram
<di@python.org>, Brett Cannon <brett@python.org> Discussions-To:
https://discuss.python.org/t/pep-650-specifying-installer-requirements-for-python-projects/6657
Status: Withdrawn Type: Standards Track Topic: Packaging Content-Type:
text/x-rst Created: 16-Jul-2020 Post-History: 14-Jan-2021

Abstract

Python package installers are not completely interoperable with each
other. While pip is the most widely used installer and a de facto
standard, other installers such as Poetry or Pipenv are popular as well
due to offering unique features which are optimal for certain workflows
and not directly in line with how pip operates.

While the abundance of installer options is good for end-users with
specific needs, the lack of interoperability between them makes it hard
to support all potential installers. Specifically, the lack of a
standard requirements file for declaring dependencies means that each
tool must be explicitly used in order to install dependencies specified
with their respective format. Otherwise tools must emit a requirements
file which leads to potential information loss for the installer as well
as an added export step as part of a developer's workflow.

By providing a standardized API that can be used to invoke a compatible
installer, we can solve this problem without needing to resolve
individual concerns, unique requirements, and incompatibilities between
different installers and their lock files.

Installers that implement the specification can be invoked in a uniform
way, allowing users to use their installer of choice as if they were
invoking it directly.

Terminology

Installer interface

    The interface by which an installer backend and a universal
    installer interact.

Universal installer

    An installer that can invoke an installer backend by calling the
    optional invocation methods of the installer interface. This can
    also be thought of as the installer frontend, à la the build project
    for PEP 517.

Installer backend

    An installer that implements the installer interface, allowing it to
    be invoked by a universal installer. An installer backend may also
    be a universal installer as well, but it is not required. In
    comparison to PEP 517, this would be Flit. Installer backends may be
    wrapper packages around a backing installer, e.g. Poetry could
    choose to not support this API, but a package could act as a wrapper
    to invoke Poetry as appropriate to use Poetry to perform an
    installation.

Dependency group

    A set of dependencies that are related and required to be installed
    simultaneously for some purpose. For example, a "test" dependency
    group could include the dependencies required to run the test suite.
    How dependency groups are specified is up to the installer backend.

Motivation

This specification allows anyone to invoke and interact with installer
backends that implement the specified interface, allowing for a
universally supported layer on top of existing tool-specific
installation processes.

This in turn would enable the use of all installers that implement the
specified interface to be used in environments that support a single
universal installer, as long as that installer implements this
specification as well.

Below, we identify various use-cases applicable to stakeholders in the
Python community and anyone who interacts with Python package
installers. For developers or companies, this PEP would allow for
increased functionality and flexibility with Python package installers.

Providers

Providers are the parties (organization, person, community, etc.) that
supply a service or software tool which interacts with Python packaging
and consequently Python package installers. Two different types of
providers are considered:

Platform/Infrastructure Providers

Platform providers (cloud environments, application hosting, etc.) and
infrastructure service providers need to support package installers for
their users to install Python dependencies. Most only support pip,
however there is user demand for other Python installers. Most providers
do not want to maintain support for more than one installer because of
the complexity it adds to their software or service and the resources it
takes to do so.

Via this specification, we can enable a provider-supported universal
installer to invoke the user-desired installer backend without the
provider’s platform needing to have specific knowledge of said backend.
What this means is if Poetry implemented the installer backend API
proposed by this PEP (or some other package wrapped Poetry to provide
the API), then platform providers would support Poetry implicitly.

IDE Providers

Integrated development environments may interact with Python package
installation and management. Most only support pip as a Python package
installer, and users are required to find work arounds to install their
dependencies using other package installers. Similar to the situation
with PaaS & IaaS providers, IDE providers do not want to maintain
support for N different Python installers. Instead, implementers of the
installer interface (installer backends) could be invoked by the IDE by
it acting as a universal installer.

Developers

Developers are teams, people, or communities that code and use Python
package installers and Python packages. Three different types of
developers are considered:

Developers using PaaS & IaaS providers

Most PaaS and IaaS providers only support one Python package installer:
pip. (Some exceptions include Heroku's Python buildpack, which supports
pip and Pipenv). This dictates the installers that developers can use
while working with these providers, which might not be optimal for their
application or workflow.

Installers adopting this PEP to become installer backends would allow
users to use third party platforms/infrastructure without having to
worry about which Python package installer they are required to use as
long as the provider uses a universal installer.

Developers using IDEs

Most IDEs only support pip or a few Python package installers.
Consequently, developers must use workarounds or hacky methods to
install their dependencies if they use an unsupported package installer.

If the IDE uses/provides a universal installer it would allow for any
installer backend that the developer wanted to be used to install
dependencies, freeing them of any extra work to install their
dependencies in order to integrate into the IDE's workflow more closely.

Developers working with other developers

Developers want to be able to use the installer of their choice while
working with other developers, but currently have to synchronize their
installer choice for compatibility of dependency installation. If all
preferred installers instead implemented the specified interface, it
would allow for cross use of installers, allowing developers to choose
an installer regardless of their collaborator’s preference.

Upgraders & Package Infrastructure Providers

Package upgraders and package infrastructure in CI/CD such as
Dependabot, PyUP, etc. currently support a few installers. They work by
parsing and editing the installer-specific dependency files directly
(such as requirements.txt or poetry.lock) with relevant package
information such as upgrades, downgrades, or new hashes. Similar to
Platform and IDE providers, most of these providers do not want to
support N different Python package installers as that would require
supporting N different file types.

Currently, these services/bots have to implement support for each
package installer individually. Inevitably, the most popular installers
are supported first, and less popular tools are often never supported.
By implementing this specification, these services/bots can support any
(compliant) installer, allowing users to select the tool of their
choice. This will allow for more innovation in the space, as platforms
and IDEs are no longer forced to prematurely select a "winner".

Open Source Community

Specifying installer requirements and adopting this PEP will reduce the
friction between Python package installers and people's workflows.
Consequently, it will reduce the friction between Python package
installers and 3rd party infrastructure/technologies such as PaaS or
IDEs. Overall, it will allow for easier development, deployment and
maintenance of Python projects as Python package installation becomes
simpler and more interoperable.

Specifying requirements and creating an interface for installers can
also increase the pace of innovation around installers. This would allow
for installers to experiment and add unique functionality without
requiring the rest of the ecosystem to do the same. Support becomes
easier and more likely for a new installer regardless of the
functionality it adds and the format in which it writes dependencies,
while reducing the developer time and resources needed to do so.

Specification

Similar to how PEP 517 specifies build systems, the install system
information will live in the pyproject.toml file under the
install-system table.

[install-system]

The install-system table is used to store install-system relevant data
and information. There are multiple required keys for this table:
requires and install-backend. The requires key holds the minimum
requirements for the installer backend to execute and which will be
installed by the universal installer. The install-backend key holds the
name of the install backend’s entry point. This will allow the universal
installer to install the requirements for the installer backend itself
to execute (not the requirements that the installer backend itself will
install) as well as invoke the installer backend.

If either of the required keys are missing or empty then the universal
installer SHOULD raise an error.

All package names interacting with this interface are assumed to follow
PEP 508's "Dependency specification for Python Software Packages"
format.

An example install-system table:

    #pyproject.toml
    [install-system]
    #Eg : pipenv
    requires = ["pipenv"]
    install-backend = "pipenv.api:main"

Installer Requirements:

The requirements specified by the requires key must be within the
constraints specified by PEP 517. Specifically, that dependency cycles
are not permitted and the universal installer SHOULD refuse to install
the dependencies if a cycle is detected.

Additional parameters or tool specific data

Additional parameters or tool (installer backend) data may also be
stored in the pyproject.toml file. This would be in the “tool.*” table
as specified by PEP 518. For example, if the installer backend is Poetry
and you wanted to specify multiple dependency groups, the tool.poetry
tables could look like this:

    [tool.poetry.dev-dependencies]
    dependencies = "dev"

    [tool.poetry.deploy]
    dependencies = "deploy"

Data may also be stored in other ways as the installer backend sees fit
(e.g. separate configuration file).

Installer interface:

The installer interface contains mandatory and optional hooks. Compliant
installer backends MUST implement the mandatory hooks and MAY implement
the optional hooks. A universal installer MAY implement any of the
installer backend hooks itself, to act as both a universal installer and
installer backend, but this is not required.

All hooks take **kwargs arbitrary parameters that a installer backend
may require that are not already specified, allowing for backwards
compatibility. If unexpected parameters are passed to the installer
backend, it should ignore them.

The following information is akin to the corresponding section in PEP
517. The hooks may be called with keyword arguments, so installer
backends implementing them should be careful to make sure that their
signatures match both the order and the names of the arguments above.

All hooks MAY print arbitrary informational text to stdout and stderr.
They MUST NOT read from stdin, and the universal installer MAY close
stdin before invoking the hooks.

The universal installer may capture stdout and/or stderr from the
backend. If the backend detects that an output stream is not a
terminal/console (e.g. not sys.stdout.isatty()), it SHOULD ensure that
any output it writes to that stream is UTF-8 encoded. The universal
installer MUST NOT fail if captured output is not valid UTF-8, but it
MAY not preserve all the information in that case (e.g. it may decode
using the replace error handler in Python). If the output stream is a
terminal, the installer backend is responsible for presenting its output
accurately, as for any program running in a terminal.

If a hook raises an exception, or causes the process to terminate, then
this indicates an error.

Mandatory hooks:

invoke_install

Installs the dependencies:

    def invoke_install(
        path: Union[str, bytes, PathLike[str]],
        *,
        dependency_group: str = None,
        **kwargs
    ) -> int:
        ...

-   path : An absolute path where the installer backend should be
    invoked from (e.g. the directory where pyproject.toml is located).
-   dependency_group : An optional flag specifying a dependency group
    that the installer backend should install. The install will error if
    the dependency group doesn't exist. A user can find all dependency
    groups by calling get_dependency_groups() if dependency groups are
    supported by the installer backend.
-   **kwargs : Arbitrary parameters that a installer backend may require
    that are not already specified, allows for backwards compatibility.
-   Returns : An exit code (int). 0 if successful, any positive integer
    if unsuccessful.

The universal installer will use the exit code to determine if the
installation is successful and SHOULD return the exit code itself.

Optional hooks:

invoke_uninstall

Uninstall the specified dependencies:

    def invoke_uninstall(
        path: Union[str, bytes, PathLike[str]],
        *,
        dependency_group: str = None,
        **kwargs
    ) -> int:
        ...

-   path : An absolute path where the installer backend should be
    invoked from (e.g. the directory where pyproject.toml is located).
-   dependency_group : An optional flag specifying a dependency group
    that the installer backend should uninstall.
-   **kwargs : Arbitrary parameters that a installer backend may require
    that are not already specified, allows for backwards compatibility.
-   Returns : An exit code (int). 0 if successful, any positive integer
    if unsuccessful.

The universal installer MUST invoke the installer backend at the same
path that the universal installer itself was invoked.

The universal installer will use the exit code to determine if the
uninstall is successful and SHOULD return the exit code itself.

get_dependencies_to_install

Returns the dependencies that would be installed by invoke_install(...).
This allows package upgraders (e.g., Dependabot) to retrieve the
dependencies attempting to be installed without parsing the dependency
file:

    def get_dependencies_to_install(
        path: Union[str, bytes, PathLike[str]],
        *,
        dependency_group: str = None,
        **kwargs
    ) -> Sequence[str]:
        ...

-   path : An absolute path where the installer backend should be
    invoked from (e.g. the directory where pyproject.toml is located).
-   dependency_group : Specify a dependency group to get the
    dependencies invoke_install(...) would install for that dependency
    group.
-   **kwargs : Arbitrary parameters that a installer backend may require
    that are not already specified, allows for backwards compatibility.
-   Returns: A list of dependencies (PEP 508 strings) to install.

If the group is specified, the installer backend MUST return the
dependencies corresponding to the provided dependency group. If the
specified group doesn't exist, or dependency groups are not supported by
the installer backend, the installer backend MUST raise an error.

If the group is not specified, and the installer backend provides the
concept of a default/unspecified group, the installer backend MAY return
the dependencies for the default/unspecified group, but otherwise MUST
raise an error.

get_dependency_groups

Returns the dependency groups available to be installed. This allows
universal installers to enumerate all dependency groups the installer
backend is aware of:

    def get_dependency_groups(
        path: Union[str, bytes, PathLike[str]],
        **kwargs
    ) -> AbstractSet[str]:
        ...

-   path : An absolute path where the installer backend should be
    invoked from (e.g. the directory where pyproject.toml is located).
-   **kwargs : Arbitrary parameters that a installer backend may require
    that are not already specified, allows for backwards compatibility.
-   Returns: A set of known dependency groups, as strings The empty set
    represents no dependency groups.

update_dependencies

Outputs a dependency file based on inputted package list:

    def update_dependencies(
        path: Union[str, bytes, PathLike[str]],
        dependency_specifiers: Iterable[str],
        *,
        dependency_group=None,
        **kwargs
    ) -> int:
        ...

-   path : An absolute path where the installer backend should be
    invoked from (e.g. the directory where pyproject.toml is located).
-   dependency_specifiers : An iterable of dependencies as PEP 508
    strings that are being updated, for example :
    ["requests==2.8.1", ...]. Optionally for a specific dependency
    group.
-   dependency_group : The dependency group that the list of packages is
    for.
-   **kwargs : Arbitrary parameters that a installer backend may require
    that are not already specified, allows for backwards compatibility.
-   Returns : An exit code (int). 0 if successful, any positive integer
    if unsuccessful.

Example

Let's consider implementing an installer backend that uses pip and its
requirements files for dependency groups. An implementation may (very
roughly) look like the following:

    import subprocess
    import sys


    def invoke_install(path, *, dependency_group=None, **kwargs):
        try:
            return subprocess.run(
                [
                    sys.executable,
                    "-m",
                    "pip",
                    "install",
                    "-r",
                    dependency_group or "requirements.txt",
                ],
                cwd=path,
            ).returncode
        except subprocess.CalledProcessError as e:
            return e.returncode

If we named this package pep650pip, then we could specify in
pyproject.toml:

    [install-system]
      #Eg : pipenv
      requires = ["pep650pip", "pip"]
      install-backend = "pep650pip:main"

Rationale

All hooks take **kwargs to allow for backwards compatibility and allow
for tool specific installer backend functionality which requires a user
to provide additional information not required by the hook.

While installer backends must be Python packages, what they do when
invoked is an implementation detail of that tool. For example, an
installer backend could act as a wrapper for a platform package manager
(e.g., apt).

The interface does not in any way try to specify how installer backends
should function. This is on purpose so that installer backends can be
allowed to innovate and solve problem in their own way. This also means
this PEP takes no stance on OS packaging as that would be an installer
backend's domain.

Defining the API in Python does mean that some Python code will
eventually need to be executed. That does not preclude non-Python
installer backends from being used, though (e.g. mamba), as they could
be executed as a subprocess from Python code.

Backwards Compatibility

This PEP would have no impact on pre-existing code and functionality as
it only adds new functionality to a universal installer. Any existing
installer should maintain its existing functionality and use cases,
therefore having no backwards compatibility issues. Only code aiming to
take advantage of this new functionality will have motivation to make
changes to their pre existing code.

Security Implications

A malicious user has no increased ability or easier access to anything
with the addition of standardized installer specifications. The
installer that could be invoked by a universal installer via the
interface specified in this PEP would be explicitly declared by the
user. If the user has chosen a malicious installer, then invoking it
with a universal installer is no different than the user invoking the
installer directly. A malicious installer being an installer backend
doesn't give it additional permissions or abilities.

Rejected Ideas

A standardized lock file

A standardized lock file would solve a lot of the same problems that
specifying installer requirements would. For example, it would allow for
PaaS/IaaS to just support one installer that could read the standardized
lock file regardless of the installer that created it. The problem with
a standardized lock file is the difference in needs between Python
package installers as well as a fundamental issue with creating
reproducible environments via the lockfile (one of the main benefits).

Needs and information stored in dependency files between installers
differ significantly and are dependent on installer functionality. For
example, a Python package installer such as Poetry requires information
for all Python versions and platforms and calculates appropriate hashes
while pip doesn't. Additionally, pip would not be able to guarantee
recreating the same environment (install the exact same dependencies) as
it is outside the scope of its functionality. This makes a standardized
lock file harder to implement and makes it seem more appropriate to make
lock files tool specific.

Have installer backends support creating virtual environments

Because installer backends will very likely have a concept of virtual
environments and how to install into them, it was briefly considered to
have them also support creating virtual environments. In the end,
though, it was considered an orthogonal idea.

Open Issues

Should the dependency_group argument take an iterable?

This would allow for specifying non-overlapping dependency groups in a
single call, e.g. "docs" and "test" groups which have independent
dependencies but which a developer may want to install simultaneously
while doing development.

Is the installer backend executed in-process?

If the installer backend is executed in-process then it greatly
simplifies knowing what environment to install for/into, as the live
Python environment can be queried for appropriate information.

Executing out-of-process allows for minimizing potential issues of
clashes between the environment being installed into and the installer
backend (and potentially universal installer).

Enforce that results from the proposed interface feed into other parts?

E.g. the results from get_dependencies_to_install() and
get_dependency_groups() can be passed into invoke_install(). This would
prevent drift between the results of various parts of the proposed
interface, but it makes more of the interface required instead of
optional.

Raising exceptions instead of exit codes for failure conditions

It has been suggested that instead of returning an exit code the API
should raise exceptions. If you view this PEP as helping to translate
current installers into installer backends, then relying on exit codes
makes sense. There's is also the point that the APIs have no specific
return value, so passing along an exit code does not interfere with what
the functions return.

Compare that to raising exceptions in case of an error. That could
potentially provide a more structured approach to error raising,
although to be able to capture errors it would require specifying
exception types as part of the interface.

References

Copyright

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.