PEP: 633 Title: Dependency specification in pyproject.toml using an
exploded TOML table Author: Laurie Opperman
<laurie_opperman@hotmail.com>, Arun Babu Neelicattu
<arun.neelicattu@gmail.com> Sponsor: Brett Cannon <brett@python.org>
Discussions-To:
https://discuss.python.org/t/dependency-specification-in-pyproject-toml-using-an-exploded-toml-table/5123/
Status: Rejected Type: Standards Track Topic: Packaging Content-Type:
text/x-rst Created: 02-Sep-2020 Post-History: 02-Sep-2020 Resolution:
https://discuss.python.org/t/how-to-specify-dependencies-pep-508-strings-or-a-table-in-toml/5243/38

Rejection Notice

This PEP has been rejected in favour of PEP 631 due to its popularity,
consistency with the existing usage of PEP 508 strings, and
compatibility with existing packaging tool suites.

Abstract

This PEP specifies how to write a project's dependencies in a
pyproject.toml file for packaging-related tools to consume using the
fields defined in PEP 621, as an alternative to the PEP 508-based
approach defined in PEP 631.

Motivation

There are multiple benefits to using TOML tables and other data-types to
represent requirements rather than PEP 508 strings:

-   Easy initial validation via the TOML syntax.
-   Easy secondary validation using a schema, for example a JSON Schema.
-   Potential for users to guess the keys of given features, rather than
    memorising a syntax.
-   Users of multiple other popular languages may already be familiar
    with the TOML syntax.
-   TOML directly represents the same data structures as in JSON, and
    therefore a sub-set of Python literals, so users can understand the
    hierarchy and type of value

Rationale

Most of this is taken from discussions in the PEP 621 dependencies
topic. This has elements from Pipfile, Poetry, Dart's dependencies and
Rust's Cargo. A comparison document shows advantages and disadvantages
between this format and PEP 508-style specifiers.

In the specification of multiple requirements with the same distribution
name (where environment markers choose the appropriate dependency), the
chosen solution is similar to Poetry's, where an array of requirements
is allowed.

The direct-reference keys closely align with and utilise PEP 610 and PEP
440 as to reduce differences in the packaging ecosystem and rely on
previous work in specification.

Specification

As in PEP 621, if metadata is improperly specified then tools MUST raise
an error. The metadata MUST conform to the TOML specification.

To reduce confusion with this document being a specification for
specifying dependencies, the word "requirement" is used to mean a PEP
508 dependency specification.

The following tables are added to the project table specified in PEP
621.

dependencies

Format: table

The keys inside this table are the names of the required distribution.
The values can have one of the following types:

-   string: the requirement is defined only by a version requirement,
    with same specification as version in the requirement table, except
    allowing the empty string "" to place no restriction on the version.
-   table: a requirement table.
-   array: an array of requirement tables. It is an error to specify an
    empty array [] as a value.

Requirement table

The keys of the requirement table are as follows (all are optional):

-   version (string): a PEP 440 version specifier, which is a
    comma-delimited list of version specifier clauses. The string MUST
    be non-empty.
-   extras (array of strings): a list of PEP 508 extras declarations for
    the distribution. The list MUST be non-empty.
-   markers (string): a PEP 508 environment marker expression. The
    string MUST be non-empty.
-   url (string): the URL of the artifact to install and satisfy the
    requirement. Note that file:// is the prefix used for packages to be
    retrieved from the local filesystem.
-   git, hg, bzr or svn (string): the URL of a VCS repository (as
    specified in PEP 440) to clone, whose tree will be installed to
    satisfy the requirement. Further VCS keys will be added via
    amendments to PEP 610, however tools MAY opt to support other VCS's
    using their command-line command prior to the acceptance of the
    amendment.
-   revision (string): the identifier for a specific revision of the
    specified VCS repository to check-out before installation. Users
    MUST only provide this when one of git, hg, bzr, svn, or another VCS
    key is used to identify the distribution to install. Revision
    identifiers are suggested in PEP 610.

At most one of the following keys can be specified simultaneously, as
they logically conflict with each other in the requirement: version,
url, git, hg, bzr, svn, and any other VCS key.

An empty requirement table {} places no restriction on the requirement,
in addition to the empty string "".

Any keys provided which are not specified in this document MUST cause an
error in parsing.

optional-dependencies

Format: table

The keys inside this table are the names of an extra's required
distribution. The values can have one of the following types:

-   table: a requirement table.
-   array: an array of requirement tables.

These requirement tables have the same specification as above, with the
addition of the following required key:

-   for-extra (string): the name of the PEP 508 extra that this
    requirement is required for.

Reference implementation

Tools will need to convert this format to PEP 508 requirement strings.
Below is an example implementation of that conversion (assuming
validation is already performed):

    def convert_requirement_to_pep508(name, requirement):
        if isinstance(requirement, str):
            requirement = {"version": requirement}
        pep508 = name
        if "extras" in requirement:
            pep508 += " [" + ", ".join(requirement["extras"]) + "]"
        if "version" in requirement:
            pep508 += " " + requirement["version"]
        if "url" in requirement:
            pep508 += " @ " + requirement["url"]
        for vcs in ("git", "hg", "bzr", "svn"):
            if vcs in requirement:
                pep508 += " @ " + vcs + "+" + requirement[vcs]
                if "revision" in requirement:
                    pep508 += "@" + requirement["revision"]
        extra = None
        if "for-extra" in requirement:
            extra = requirement["for-extra"]
        if "markers" in requirement:
            markers = requirement["markers"]
            if extra:
                markers = "extra = '" + extra + "' and (" + markers + ")"
            pep508 += "; " + markers
        return pep508, extra


    def convert_requirements_to_pep508(dependencies):
        pep508s = []
        extras = set()
        for name, req in dependencies.items():
            if isinstance(req, list):
                for sub_req in req:
                    pep508, extra = convert_requirement_to_pep508(name, sub_req)
                    pep508s.append(pep508)
                    if extra:
                        extras.add(extra)
            else:
                pep508, extra = convert_requirement_to_pep508(name, req)
                pep508s.append(pep508)
                if extra:
                    extras.add(extra)
        return pep508s, extras


    def convert_project_requirements_to_pep508(project):
        reqs, _ = convert_requirements_to_pep508(project.get("dependencies", {}))
        optional_reqs, extras = convert_requirements_to_pep508(
            project.get("optional-dependencies", {})
        )
        reqs += optional_reqs
        return reqs, extras

JSON schema

For initial validation, a JSON-schema can be used. Not only does this
help tools have a consistent validation, but it allows code editors to
highlight validation errors as users are building the dependencies list.

    {
        "$id": "spam",
        "$schema": "http://json-schema.org/draft-07/schema#",
        "title": "Project metadata",
        "type": "object",
        "definitions": {
            "requirementTable": {
                "title": "Full project dependency specification",
                "type": "object",
                "properties": {
                    "extras": {
                        "title": "Dependency extras",
                        "type": "array",
                        "items": {
                            "title": "Dependency extra",
                            "type": "string"
                        }
                    },
                    "markers": {
                        "title": "Dependency environment markers",
                        "type": "string"
                    }
                },
                "propertyNames": {
                    "enum": [
                        "extras",
                        "markers",
                        "version",
                        "url",
                        "git",
                        "hg",
                        "bzr",
                        "svn",
                        "for-extra"
                    ]
                },
                "oneOf": [
                    {
                        "title": "Version requirement",
                        "properties": {
                            "version": {
                                "title": "Version",
                                "type": "string"
                            }
                        }
                    },
                    {
                        "title": "URL requirement",
                        "properties": {
                            "url": {
                                "title": "URL",
                                "type": "string",
                                "format": "uri"
                            }
                        },
                        "required": [
                            "url"
                        ]
                    },
                    {
                        "title": "VCS requirement",
                        "properties": {
                            "revision": {
                                "title": "VCS repository revision",
                                "type": "string"
                            }
                        },
                        "oneOf": [
                            {
                                "title": "Git repository",
                                "properties": {
                                    "git": {
                                        "title": "Git URL",
                                        "type": "string",
                                        "format": "uri"
                                    }
                                },
                                "required": [
                                    "git"
                                ]
                            },
                            {
                                "title": "Mercurial repository",
                                "properties": {
                                    "hg": {
                                        "title": "Mercurial URL",
                                        "type": "string",
                                        "format": "uri"
                                    }
                                },
                                "required": [
                                    "hg"
                                ]
                            },
                            {
                                "title": "Bazaar repository",
                                "properties": {
                                    "bzr": {
                                        "title": "Bazaar URL",
                                        "type": "string",
                                        "format": "uri"
                                    }
                                },
                                "required": [
                                    "bzr"
                                ]
                            },
                            {
                                "title": "Subversion repository",
                                "properties": {
                                    "svn": {
                                        "title": "Subversion URL",
                                        "type": "string",
                                        "format": "uri"
                                    }
                                },
                                "required": [
                                    "svn"
                                ]
                            }
                        ]
                    }
                ]
            },
            "requirementVersion": {
                "title": "Version project dependency specification",
                "type": "string"
            },
            "requirement": {
                "title": "Project dependency specification",
                "oneOf": [
                    {
                        "$ref": "#/definitions/requirementVersion"
                    },
                    {
                        "$ref": "#/definitions/requirementTable"
                    },
                    {
                        "title": "Multiple specifications",
                        "type": "array",
                        "items": {
                            "$ref": "#/definitions/requirementTable"
                        },
                        "minLength": 1
                    }
                ]
            },
            "optionalRequirementTable": {
                "title": "Project optional dependency specification table",
                "allOf": [
                    {
                        "$ref": "#/definitions/requirementTable"
                    },
                    {
                        "properties": {
                            "for-extra": {
                                "title": "Dependency's extra",
                                "type": "string"
                            }
                        },
                        "required": [
                            "for-extra"
                        ]
                    }
                ]
            },
            "optionalRequirement": {
                "title": "Project optional dependency specification",
                "oneOf": [
                    {
                        "$ref": "#/definitions/optionalRequirementTable"
                    },
                    {
                        "title": "Multiple specifications",
                        "type": "array",
                        "items": {
                            "$ref": "#/definitions/optionalRequirementTable"
                        },
                        "minLength": 1
                    }
                ]
            }
        },
        "properties": {
            "dependencies": {
                "title": "Project dependencies",
                "type": "object",
                "additionalProperties": {
                    "$ref": "#/definitions/requirement"
                }
            },
            "optional-dependencies": {
                "title": "Project dependencies",
                "type": "object",
                "additionalProperties": {
                    "$ref": "#/definitions/optionalRequirement"
                }
            }
        }
    }

Examples

Full artificial example:

    [project.dependencies]
    flask = { }
    django = { }
    requests = { version = ">= 2.8.1, == 2.8.*", extras = ["security", "tests"], markers = "python_version < '2.7'" }
    pip = { url = "https://github.com/pypa/pip/archive/1.3.1.zip" }
    sphinx = { git = "ssh://git@github.com/sphinx-doc/sphinx.git" }
    numpy = "~=1.18"
    pytest = [
        { version = "<6", markers = "python_version < '3.5'" },
        { version = ">=6", markers = "python_version >= '3.5'" },
    ]

    [project.optional-dependencies]
    pytest-timout = { for-extra = "dev" }
    pytest-mock = [
        { version = "<6", markers = "python_version < '3.5'", for-extra = "dev" },
        { version = ">=6", markers = "python_version >= '3.5'", for-extra = "dev" },
    ]

In homage to PEP 631, the following is an equivalent dependencies
specification for docker-compose:

    [project.dependencies]
    cached-property = ">= 1.2.0, < 2"
    distro = ">= 1.2.0, < 2"
    docker = { extras = ["ssh"], version = ">= 4.2.2, < 5" }
    docopt = ">= 0.6.1, < 1"
    jsonschema = ">= 2.5.1, < 4"
    PyYAML = ">= 3.10, < 6"
    python-dotenv = ">= 0.13.0, < 1"
    requests = ">= 2.20.0, < 3"
    texttable = ">= 0.9.0, < 2"
    websocket-client = ">= 0.32.0, < 1"

    # Conditional
    "backports.shutil_get_terminal_size" = { version = "== 1.0.0", markers = "python_version < '3.3'" }
    "backports.ssl_match_hostname" = { version = ">= 3.5, < 4", markers = "python_version < '3.5'" }
    colorama = { version = ">= 0.4, < 1", markers = "sys_platform == 'win32'" }
    enum34 = { version = ">= 1.0.4, < 2", markers = "python_version < '3.4'" }
    ipaddress = { version = ">= 1.0.16, < 2", markers = "python_version < '3.3'" }
    subprocess32 = { version = ">= 3.5.4, < 4", markers = "python_version < '3.2'" }

    [project.optional-dependencies]
    PySocks = { version = ">= 1.5.6, != 1.5.7, < 2", for-extra = "socks" }
    ddt = { version = ">= 1.2.2, < 2", for-extra = "tests" }
    pytest = { version = "< 6", for-extra = "tests" }
    mock = { version = ">= 1.0.1, < 4", markers = "python_version < '3.4'", for-extra = "tests" }

Compatibility Examples

The authors of this PEP recognise that various tools need to both read
from and write to this format for dependency specification. This section
aims to provide direct comparison with and examples for translating
to/from the currently used standard, PEP 508.

Note

For simplicity and clarity, various ways in which TOML allows you to
specify each specification is not represented. These examples use the
standard inline representation.

For example, while following are considered equivalent in TOML, we
choose the second form for the examples in this section.

    aiohttp.version = "== 3.6.2"
    aiohttp = { version = "== 3.6.2" }

Version Constrained Dependencies

No Version Constraint

    aiohttp

    aiohttp = {}

Simple Version Constraint

    aiohttp >= 3.6.2, < 4.0.0

    aiohttp = { version = ">= 3.6.2, < 4.0.0" }

Note

This can, for conciseness, be also represented as a string.

    aiohttp = ">= 3.6.2, < 4.0.0"

Direct Reference Dependencies

URL Dependency

    aiohttp @ https://files.pythonhosted.org/packages/97/d1/1cc7a1f84097d7abdc6c09ee8d2260366f081f8e82da36ebb22a25cdda9f/aiohttp-3.6.2-cp35-cp35m-macosx_10_13_x86_64.whl

    aiohttp = { url = "https://files.pythonhosted.org/packages/97/d1/1cc7a1f84097d7abdc6c09ee8d2260366f081f8e82da36ebb22a25cdda9f/aiohttp-3.6.2-cp35-cp35m-macosx_10_13_x86_64.whl" }

VCS Dependency

    aiohttp @ git+ssh://git@github.com/aio-libs/aiohttp.git@master

    aiohttp = { git = "ssh://git@github.com/aio-libs/aiohttp.git", revision = "master" }

Environment Markers

    aiohttp >= 3.6.1; python_version >= '3.8'

    aiohttp = { version = ">= 3.6.1", markers = "python_version >= '3.8'" }

A slightly extended example of the above, where a particular version of
aiohttp is required based on the interpreter version.

    aiohttp >= 3.6.1; python_version >= '3.8'
    aiohttp >= 3.0.0, < 3.6.1; python_version < '3.8'

    aiohttp = [
        { version = ">= 3.6.1", markers = "python_version >= '3.8'" },
        { version = ">= 3.0.0, < 3.6.1", markers = "python_version < '3.8'" }
    ]

Package Extras

Specifying dependency for a package extra

    aiohttp >= 3.6.2; extra == 'http'

    aiohttp = { version = ">= 3.6.2", for-extra = "http" }

Using extras from a dependency

    aiohttp [speedups] >= 3.6.2

    aiohttp = { version = ">= 3.6.2", extras = ["speedups"] }

Complex Examples

Version Constraint

    aiohttp [speedups] >= 3.6.2; python_version >= '3.8' and extra == 'http'

    aiohttp = { version = ">= 3.6.2", extras = ["speedups"], markers = "python_version >= '3.8'", for-extra = "http" }

Direct Reference (VCS)

    aiohttp [speedups] @ git+ssh://git@github.com/aio-libs/aiohttp.git@master ; python_version >= '3.8' and extra == 'http'

    aiohttp = { git = "ssh://git@github.com/aio-libs/aiohttp.git", revision = "master", extras = ["speedups"], markers = "python_version >= '3.8'", for-extra = "http" }

Rejected Ideas

Switch to an array for dependencies

Use an array instead of a table in order to have each element only be a
table (with a name key) and no arrays of requirement tables. This was
very verbose and restrictive in the TOML format, and having multiple
requirements for a given distribution isn't very common.

Replace optional-dependencies with extras

Remove the optional-dependencies table in favour of both including an
optional key in the requirement and an extras table which specifies
which (optional) requirements are needed for a project's extra. This
reduces the number of table with the same specification (to 1) and
allows for requirements to be specified once but used in multiple
extras, but distances some of the requirement's properties (which
extra(s) it belongs to), groups required and optional dependencies
together (possibly mixed), and there may not be a simple way to choose a
requirement when a distribution has multiple requirements. This was
rejected as optional-dependencies has already been used in the PEP 621
draft.

direct table in requirement

Include the direct-reference keys in a direct table, have the VCS
specified as the value of a vcs key. This was more explicit and easier
to include in a JSON-schema validation, but was decided to be too
verbose and not as readable.

Include hash

Include hash in direct-reference requirements. This was only for package
lock-files, and didn't really have a place in the project's metadata.

Dependency tables for each extra

Have the optional-dependencies be a table of dependency tables for each
extra, with the table name being the extra's name. This made
optional-dependencies a different type (table of tables of requirements)
from dependencies (table of requirements), which could be jarring for
users and harder to parse.

Environment marker keys

Make each PEP 508 environment marker as a key (or child-table key) in
the requirement. This arguably increases readability and ease of
parsing. The markers key would still be allowed for more advanced
specification, with which the key-specified environment markers are
and'd with the result of. This was deferred as more design needs to be
undertaken.

Multiple extras which one requirement can satisfy

Replace the for-extra key with for-extras, with the value being an array
of extras which the requirement satisfies. This reduces some
duplication, but in this case that duplication makes explicit which
extras have which dependencies.

Copyright

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.