PEP: 700 Title: Additional Fields for the Simple API for Package Indexes
Author: Paul Moore <p.f.moore@gmail.com> PEP-Delegate: Donald Stufft
<donald@stufft.io> Discussions-To:
https://discuss.python.org/t/pep-700-additional-fields-for-the-simple-api-for-package-indexes/20177
Status: Final Type: Standards Track Topic: Packaging Created:
21-Oct-2022 Post-History: 21-Oct-2022 Resolution: 19-Dec-2022

packaging:simple-repository-api

Abstract

PEP 691 defined a JSON form for the "Simple Repository API". This
allowed clients to more easily query the data that was previously only
available in HTML, as defined in PEP 503.

This proposal adds three fields to the JSON form, which allow it to be
used in place of PyPI's JSON API in a number of situations.

-   A field to allow retrieval of a list of all the published versions
    of a project.
-   Fields containing the size and upload time for a project file.

The new fields are all part of the data returned from the "project
details" URL.

Rationale

With the introduction of the JSON form of the simple API in PEP 691, the
simple API offers functionality that is almost as complete as the PyPI
JSON API. This PEP adds a number of fields which were previously only
available through the JSON API, in order to allow more clients which
were previously Warehouse specific to support arbitrary
standards-compliant indexes.

Specification

This specification defines version 1.1 of the simple repository API. For
the HTML version of the API, there is no change from version 1.0. For
the JSON version of the API, the following changes are made:

-   The api-version must specify version 1.1 or later.
-   A new versions key is added at the top level.
-   Two new "file information" keys, size and upload-time, are added to
    the files data.
-   Keys (at any level) with a leading underscore are reserved as
    private for index server use. No future standard will assign a
    meaning to any such key.

The versions and size keys are mandatory. The upload-time key is
optional.

Versions

An additional key, versions MUST be present at the top level, in
addition to the keys name, files and meta defined in PEP 691. This key
MUST contain a list of version strings specifying all of the project
versions uploaded for this project. The value is logically a set, and as
such may not contain duplicates, and the order of the values is not
significant.

All of the files listed in the files key MUST be associated with one of
the versions in the versions key. The versions key MAY contain versions
with no associated files (to represent versions with no files uploaded,
if the server has such a concept).

Note that because servers may hold "legacy" data from before the
adoption of PEP 440, version strings currently cannot be required to be
valid PEP 440 versions, and therefore cannot be assumed to be orderable
using the PEP 440 rules. However, servers SHOULD use normalised PEP 440
versions where possible.

Additional file information

Two new keys are added to the files key.

-   size: This field is mandatory. It MUST contain an integer which is
    the file size in bytes.
-   upload-time: This field is optional. If present, it MUST contain a
    valid ISO 8601 date/time string, in the format
    yyyy-mm-ddThh:mm:ss.ffffffZ, which represents the time the file was
    uploaded to the index. As indicated by the Z suffix, the upload time
    MUST use the UTC timezone. The fractional seconds part of the
    timestamp (the .ffffff part) is optional, and if present may contain
    up to 6 digits of precision. If a server does not record upload time
    information for a file, it MAY omit the upload-time key.

FAQ

Why not add this data to the HTML API as well?

It would be possible to add the data to the HTML API, but the vast
majority of consumers for this data are likely to be currently getting
it from the PyPI JSON API, and so will already be expecting to parse
JSON. Traditional consumers of the HTML API have never needed this data
previously.

Does this imply that the HTML API is obsolete?

No. The FAQ of PEP 691 was clear that the HTML API is not being
deprecated, and this PEP does not change that position. However, clients
wishing to access the new data introduced by this PEP will need to use
the JSON API to get it. And indexes wanting to provide it will need to
serve the JSON format.

Is the simple API replacing the Warehouse JSON and XML-RPC APIs?

Where possible, clients should prefer the simple API over the JSON or
XML-RPC APIs, as the former is standardised and can be assumed to be
available from any index, whereas the latter are exclusive to the
Warehouse project.

However, while this PEP brings the simple API closer to being able to
replace the JSON API, there is no formal policy that the simple API will
replicate all of the functionality covered by the existing Warehouse
APIs. Proposed additions to the simple API will still be considered on
their individual merits, and the requirement that the API should be
simple and fast for the primary use case of locating files for a project
will remain the overriding consideration.

Why not allow other date formats?

The ISO 8601 standard is complex, and there seems little value in
requiring clients to deal with that. The standard library datetime
module provides methods to parse ISO 8601 strings, but it is possible
that users may want to access index data without using Python (for
example, piping the output of curl into jq). Having a single,
well-defined format makes this easier, and doesn't have any significant
disadvantages.

What if file sizes are too big for a JSON number?

The JSON standard does not specify how numbers are to be interpreted.
Python can read and write arbitrary-length integers in a JSON file, so
this should not be an issue for code written in Python. Non-Python
implementations may need to take care to handle large integers
correctly, but this is not expected to be a significant problem.

Why not require PEP 440 versions?

At the time this PEP was written, PyPI still contains (and serves)
projects and files with "legacy" versions. Requiring PEP 440 versions
would make it impossible for PyPI to follow this specification while
still serving the existing content.

Ideally, at some future time, the simple index API will be updated to
require PEP 440 versions, at which time this specification should be
updated to reflect that. However, that change will need to be
co-ordinated with existing index providers including PyPI, to desupport
and remove non-conforming projects and/or files.

Why not provide a "latest version" value?

For PEP 440 versions, this is easy enough for the client to do (using
the packaging library,
latest = max(Version(s) for s in proj["versions"])). For non-standard
versions, there is no well-defined ordering, and clients will need to
decide on what rule is appropriate for their needs. Requiring the server
to supply a latest version value takes the choice away from the client.

Servers with an explicit concept of which version is the "latest", which
cannot be calculated from data available to the client, can provide a
non-standard, underscore-prefixed key to convey that information to the
client if they wish.

Copyright

This document is placed in the public domain or under the
CC0-1.0-Universal license, whichever is more permissive.