PEP: 598 Title: Introducing incremental feature releases Version:
$Revision$ Last-Modified: $Date$ Author: Alyssa Coghlan
<ncoghlan@gmail.com> Discussions-To:
https://discuss.python.org/t/pep-596-python-3-9-release-schedule-doubling-the-release-cadence/1828
Status: Withdrawn Type: Informational Content-Type: text/x-rst Created:
15-Jun-2019 Python-Version: 3.9

Abstract

PEP 602 proposes reducing the feature delivery latency for the Python
standard library and CPython reference interpreter by increasing the
frequency of CPython feature releases from every 18-24 months to instead
occur every 9-12 months.

This PEP proposes to instead reduce the frequency of new baseline
feature releases (with the associated filesystem layout changes,
bytecode format changes, and C ABI compatibility breaks) to occur only
every other year (2020, 2022, 2024, etc), but to combine that change
with a new policy and approach that allows the introduction of backwards
compatible features in the initial set of point releases within a given
release series.

PEP Withdrawal

This PEP has been withdrawn in favour of the rolling beta release stream
proposal in PEP 605.

However, the concerns raised in this PEP are likely to apply to any
other "Long Term Support branch" proposals that allow feature backports
to improve the developer experience of supporting such releases (such as
the EL Python draft at[1]), so the ideas presented here may provide
useful design suggestions for such proposals.

Summary

The proposal to keep the current CPython release compatibility
management process, but go through it more often has significant
practical downsides, as a CPython feature release carries certain
expectations (most notably, a 5-year maintenance lifecycle, support for
parallel installation with the previous feature release, and the
possibility of breaking changes to the CPython-specific ABI, requiring
recompilation of all extension modules) that mean faster feature
releases in their current form have the potential to significantly
increase the burden of maintaining 3rd party Python libraries and
applications across all actively supported CPython releases.

It's also arguable whether such an approach would noticeably reduce the
typical feature delivery latency in practice for most end users, as the
adoption cycle for new feature releases is typically measured in months
or years, so more frequent releases may just lead to end users updating
to every 3rd or 4th feature release, rather than every 2nd or 3rd
feature release (as often happens today).

This PEP presents a competing proposal to instead slow down the
frequency of parallel installable feature releases that change the
filesystem layout and CPython ABI to a consistent 24-month cycle, but to
compensate for this by introducing the notion of build compatible
incremental feature releases, and then deferring the full feature freeze
of a given feature release series from the initial baseline X.Y.0
release to a subsequent X.Y.Z feature complete release that occurs ~12
months after the initial baseline feature release.

A new feature_complete attribute on the sys.version_info structure will
provide a programmatic indicator as to whether or not a release series
remains open to further incremental feature releases. Alternate
implementations of Python would also be free to clear this flag to
indicate that their support for their nominal Python version may still
be a work in progress.

For compatibility testing purposes, and to maintain pickle compatibility
in mixed version environments, a new sys.feature_limit attribute (and
corresponding CPython CLI parameter, --feature-limit X.Y.Z, and
environment variable, PYTHONFEATURELIMIT) will provide a way to limit
the runtime availability of features added in incremental feature
releases.

The existing cycle and the new cycle would be synchronised on their
feature freeze release dates, so the full Python 3.9.x feature freeze
would occur in October 2021, 24 months after the Python 3.8.0 feature
release, with the initial Python 3.9.0 release taking place in October
2020.

Example Future Release Schedules

Under this proposal, Python 3.9.0a1 would be released in November 2019,
shortly after the Python 3.8.0 feature complete release in October 2019.

The 3.9.0b1 release would then follow 6 months later in May 2020, with
3.9.0 itself being released in October 2020.

Assuming micro releases of 3.9.x were to occur quarterly, then the
overall release timeline would look like:

-   2019-11: 3.9.0a1
-   ... additional alpha releases as determined by the release manager
-   2020-05: 3.9.0b1
-   ... additional beta releases as determined by the release manager
-   2020-08: 3.9.0bX (final beta release that locks ABI compatibility)
-   2020-09: 3.9.0rc1
-   ... additional release candidates as determined by the release
    manager
-   2020-10: 3.9.0 (BFR - baseline feature release)
-   2021-01: 3.9.1 (IFR - incremental feature release)
-   2021-04: 3.9.2 (IFR)
-   2021-07: 3.9.3 (IFR)
-   2021-10: 3.9.4 (feature complete release)
-   2022-01: 3.9.5
-   2022-04: 3.9.6
-   2022-07: 3.9.7
-   2022-10: 3.9.8 (final regular maintenance release)
-   ... additional security fix only releases as needed
-   2025-10: 3.9.x branch closed

Feature complete release numbers would typically be written without any
qualifier (as they are today), while the baseline and incremental
feature releases would be expected to have a qualifier appended
indicating that they aren't a traditional CPython release (3.9.0 (BFR),
3.9.1 (IFR), etc).

The Python 3.10 release series would start being published the month
after the first Python 3.9 feature complete release, in parallel with
the final 12 months of routine Python 3.9 maintenance releases:

-   2021-11: 3.10.0a1
-   ... additional alpha releases as determined by the release manager
-   2022-05: 3.10.0b1
-   ... additional beta releases as determined by the release manager
-   2022-08: 3.10.0bX (final beta release that locks ABI compatibility)
-   2022-09: 3.10.0rc1
-   ... additional release candidates as determined by the release
    manager
-   2022-10: 3.10.0 (BFR)
-   2023-01: 3.10.1 (IFR)
-   2023-04: 3.10.2 (IFR)
-   2023-07: 3.10.3 (IFR)
-   2023-10: 3.10.4
-   2024-01: 3.10.5
-   2024-04: 3.10.6
-   2024-07: 3.10.7
-   2024-10: 3.10.8 (final regular maintenance release)
-   ... additional security fix only releases as needed
-   2027-10: 3.10.x branch closed

In this model, there are always two or three active branches:

-   2019-04 -> 2019-10: 3.9.0 pre-alpha, 3.8.0 pre-release, 3.7.x
    maintenance
-   2019-10 -> 2020-05: 3.9.0 pre-beta, 3.8.x maintenance
-   2020-05 -> 2020-10: 3.10.0 pre-alpha, 3.9.0 pre-release, 3.8.x
    maintenance
-   2020-10 -> 2021-10: 3.10.0 pre-alpha, 3.9.x feature releases, 3.8.x
    maintenance
-   2021-10 -> 2022-05: 3.10.0 pre-beta, 3.9.x maintenance
-   2022-05 -> 2022-10: 3.11.0 pre-alpha, 3.10.0 pre-release, 3.9.x
    maintenance
-   2022-10 -> 2023-10: 3.11.0 pre-alpha, 3.10.x feature releases, 3.9.x
    maintenance
-   2023-10 -> 2024-05: 3.11.0 pre-beta, 3.10.x maintenance
-   2024-05 -> 2024-10: 3.12.0 pre-alpha, 3.11.0 pre-release, 3.10.x
    maintenance
-   ... etc

(Pre-alpha and pre-beta development occurs on the main git branch, all
other development occurs on a release specific branch with changes
typically backported from the main git branch)

TODO: this really needs a diagram to help explain it, so I'll add a
picture once I have one to add.

This is quite similar to the status quo, but with a more consistent
cadence, alternating between baseline feature release years (2020, 2022,
etc) that focus on the alpha and beta cycle for a new baseline feature
release (while continuing to publish maintenance releases for the
previous feature release series), and feature complete release years
(2021, 2023, etc), that focus on making smaller improvements to the
current feature release series (while making plans for the next feature
release series the following year).

Proposal

Excluding alpha and beta releases, CPython currently has 3 different
kinds of release increment:

-   Feature release (i.e. X.Y.0 releases)
-   Maintenance release (X.Y.Z releases within ~2 years of X.Y.0)
-   Source-only security release (subsequent X.Y.Z releases)

Feature freeze takes place at the time of the X.Y.0b1 release. Build
compatibility freeze now takes place at the time of the last beta
release (providing time for projects to upload wheel archives to PyPI
prior to the first release candidate).

This then creates the following periods in the lifecycle of a release
series:

-   Pre-beta (release series is the CPython development branch)
-   Beta (release enters maintenance mode, ABI compatibility mostly
    locked)
-   Maintenance (ABI locked, only bug fixes & docs enhancements
    accepted)
-   Security fix only (no further binary releases, only security fixes
    accepted)
-   End of life (no further releases of any kind)

The proposal in this PEP is that the "Feature release" category be split
up into three different kinds of feature release:

-   Baseline feature release (X.Y.0 releases)
-   Incremental feature release (any X.Y.Z releases published between a
    baseline feature release and the corresponding feature complete
    release)
-   Feature complete release (a specific X.Y.Z release ~1 year after
    X.Y.0)
-   Maintenance release (X.Y.Z releases within ~1 years of the feature
    complete release)
-   Source-only security release (subsequent X.Y.Z releases)

This would then introduce a new "Feature releases" phase in the release
series lifecycle:

-   Pre-beta (release series is the CPython development branch)
-   Beta (release enters feature additions mode, ABI compatibility not
    yet locked)
-   Feature releases (ABI locked, backwards compatible API additions
    accepted)
-   Maintenance (ABI locked, only bug fixes & docs enhancements
    accepted)
-   Security fix only (no further binary releases, only security fixes
    accepted)
-   End of life (no further releases of any kind)

The pre-release beta period would be relaxed to use the incremental
feature release policy for changes, rather than the stricter maintenance
release policy.

For governance purposes, baseline feature releases are the only releases
that would qualify as a "feature release" in the PEP 13 sense
(incremental feature releases wouldn't count).

Baseline feature releases and feature release series

Baseline feature releases are essentially just the existing feature
releases, given a new name to help distinguish them from the new
incremental feature releases, and also to help indicate that unlike
their predecessors, they are no longer considered feature complete at
release.

Baseline feature releases would continue to define a new feature release
series, locking in the following language, build, and installation
compatibility constraints for the remainder of that series:

-   Python language grammar
-   ast module AST format
-   CPython interpreter opcode format
-   pyc file magic number and filename compatibility tags
-   extension module filename compatibility tags
-   wheel archive compatibility tags
-   default package and module import directories
-   default installation filename and directories

Baseline feature releases would also continue to be the only releases
where:

-   new deprecations, pending deprecations, and other warnings can be
    introduced
-   existing pending deprecations can be converted to full deprecations
-   existing warnings can be converted to errors
-   other changes requiring "Porting to Python X.Y" entries in the
    What's New document can be introduced

Key characteristics of a feature release series:

-   an installation within one feature release series does not conflict
    with installations of other feature release series (i.e. they can be
    installed in parallel)
-   an installation within a feature release series can be updated to a
    later micro release within the same series without requiring
    reinstallation or any other changes to previously installed
    components

Key characteristics of a baseline feature release:

-   in a baseline feature release,
    sys.version_info.feature_complete == False
-   in a baseline feature release, sys.version_info.micro == 0
-   baseline feature releases may contain higher risk changes to the
    language and interpreter, such as grammar modifications, major
    refactoring of interpreter and standard library internals, or
    potentially invasive feature additions that carry a risk of
    unintended side effects on other existing functionality
-   features introduced in a baseline feature release are the only
    features permitted to rely on sys.version_info as their sole runtime
    indicator of the feature's availability

Key expectations around feature release series and baseline feature
releases:

-   most public projects will only actively test against the most recent
    micro release within a release series
-   many (most?) public projects will only add a new release series to
    their test matrix after the initial baseline feature release has
    already been published, which can make it difficult to resolve
    issues that require providing new flags or APIs to explicitly opt-in
    to old behaviour after a default behaviour changed
-   private projects with known target environments will test against
    whichever micro release version they're actually using
-   most private projects will also only consider migrating to a new
    release series after the initial baseline feature release has
    already been published, again posing a problem if the resolution of
    their problems requires an API addition

The key motivation of the proposal in this PEP is that the public and
private project behaviours described above aren't new expectations:
they're descriptions of the way CPython release series are already
handled by the wider community today. As such, the PEP represents an
attempt to adjust our release policies and processes to better match the
way the wider community already handles them, rather than changing our
processes in a way that then means the wider community needs to adjust
to us rather than the other way around.

Incremental feature releases

Incremental feature releases are the key new process addition being
proposed by this PEP. They are subject to the same strict runtime
compatibility requirements as the existing maintenance releases, but
would have the following more relaxed policies around API additions and
enhancements:

-   new public APIs can be added to any standard library module
    (including builtins)
-   subject to the feature detection requirement below, new optional
    arguments can be added to existing APIs (including builtins)
-   new public APIs can be added to the stable C ABI (with appropriate
    version guards)
-   new public APIs can be added to the CPython C API
-   with the approval of the release manager, backwards compatible
    reliability improvements can be made to existing APIs and syntactic
    constructs
-   with the approval of the release manager, performance improvements
    can be incorporated for existing APIs and syntactic constructs

The intent of this change in policy is to allow usability improvements
for new (and existing!) language features to be delivered in a more
timely fashion, rather than requiring users to incur the inherent delay
and costs of waiting for and then upgrading to the next feature release
series.

It is also designed such that the approval to add a feature to the next
baseline feature release can be considered separately from the question
of whether or not to make it available in the next incremental feature
release for the current release series, potentially allowing the first
task to be completed by volunteer contributors, while the latter
activity could be handled by paid contributors (e.g. customers of
commercial Python redistributors could potentially request that their
vendor backport a feature, or core developers could offer to undertake
specific backports on a contract basis). (There would be potential
ethical concerns with gating bug fixes this way, but those concerns
don't apply for backports of new features)

Key characteristics of an incremental feature release:

-   in an incremental feature release,
    sys.version_info.feature_complete == False
-   in an incremental feature release, sys.version_info.micro != 0
-   all API additions made in an incremental feature release must
    support efficient runtime feature detection that doesn't rely on
    either sys.version_info or runtime code object introspection. In
    most cases, a simple hasattr check on the affected module will serve
    this purpose, but when it doesn't, an alternative approach will need
    to be implemented as part of the feature addition. Prior art in this
    area includes the pickle.HIGHEST_PROTOCOL attribute, the
    hashlib.algorithms_available set, and the various os.supports_* sets
    that the os module already offers for platform dependent capability
    detection
-   to maintain pickle compatibility in mixed version environments, and
    to enable easier compatibility testing across multiple API versions
    within the same release series, all API additions made in an
    incremental feature release must support the new sys.feature_limit
    setting as described in the next section

Key expectations around incremental feature releases:

-   "don't break existing installations on upgrade" remains a key
    requirement for all micro releases, even with the more permissive
    change inclusion policy
-   more intrusive changes should still be deferred to the next baseline
    feature release
-   public Python projects that start relying on features added in an
    incremental feature release should set their Python-Requires
    metadata appropriately (projects already do this when necessary -
    e.g. aiohttp specifically requires 3.5.3 or later due to an issue
    with asyncio.get_event_loop() in earlier versions)

Some standard library modules may also impose their own restrictions on
acceptable changes in incremental feature releases (for example, only a
baseline feature release should ever add new hash algorithms to
hashlib.algorithms_guaranteed - incremental feature releases would only
be permitted to add algorithms to hashlib.algorithms_available)

Maintaining interoperability across incremental feature releases

It is a common practice to use Python's pickle module to exchange
information between Python processes running on different versions of
Python. Between release series, this compatibility is expected to only
run one way (i.e. excluding deprecated APIs, Python "X.Y+1" processes
should be able to read pickle archives produced by Python "X.Y"
processes, but the reverse does not hold, as the newer archives may
reference attributes and parameters that don't exist in the older
version).

Within a release series, however, it is expected to hold in both
directions, as the "No new features" policy means that almost all pickle
archives created on Python "X.Y.Z+1" will be readable by Python "X.Y.Z"
processes.

Similarly, Python libraries and applications are often only tested
against the latest version in a release series, and this is usually
sufficient to keep code working on earlier releases in that same series.

Allowing feature additions in later "X.Y.Z" releases with no way to turn
them off would pose a problem for these common practices, as a library
or application that works fine when tested on CPython version "X.Y.Z"
would fail on earlier versions if it used a feature newly introduced in
"X.Y.Z", and any pickle archives it creates that rely on those new
interfaces may also not be readable on the older versions.

To help address these problems, a new sys.feature_limit attribute would
be added, as a structured sequence corresponding to the first 3 fields
in sys.version_info (major, minor, micro).

A new CLI option (--feature-limit X.Y.Z) and environment variable
(PYTHONFEATURELIMIT=X.Y.Z) would be used to set this attribute. The
PyCoreConfig struct would also gain a new field:

    wchar_t *feature_limit;

If the limit is not set explicitly, it would default to the first 3
fields in sys.version_info. If the limit is set to a value outside the
lower bound of sys.version_info[:2] and the upper bound of
sys.version_info[:3], it will be clamped to those bounds, padding with
zeroes if necessary.

For example, given a current version of "3.9.3", nominal limits would be
converted to runtime sys.feature_limit values as follows:

    3 => (3, 9, 0)
    3.8.1 => (3, 9, 0)
    3.9 => (3, 9, 0)
    3.9.2 => (3, 9, 2)
    <unset> => (3, 9, 3)
    3.9.3 => (3, 9, 3)
    3.9.4 => (3, 9, 3)
    4 => (3, 9, 3)

New APIs backported to an incremental feature release would be expected
to include a guard that deletes the API from the module if the feature
limit is too low:

    def feature_api():
        ...

    _version_feature_api_added = (3, 9, 1)
    if _version_feature_api_added > sys.feature_limit:
        del feature_api

Similarly, new parameters would be expected to include a guard that
adjusts the function signature to match the old one:

    def feature_api(old_param1, old_param2, new_param=default):
        """Updated API docstring"""
        ...

    _version_feature_api_changed = (3, 9, 1)
    if _version_feature_api_changed > sys.feature_limit:
        _new_feature_api = feature_api
        def feature_api(old_param1, old_param2):
            """Legacy API docstring"""
            return _new_feature_api(old_param1, old_param2)

Structuring the guards this way would keep the code structure as similar
as possible between the main development branch and the backport
branches, so future bug fixes can still be backported automatically.

It is expected that convenience functions and/or additional automated
tests would eventually be added to help ensure these backported APIs are
guarded appropriately, but it seems reasonable to wait until specific
concrete examples are available to drive the design of those APIs and
automated tests, rather than designing them solely on the basis of
hypothetical examples.

Feature complete release and subsequent maintenance releases

The feature complete release for a given feature release series would be
developed under the normal policy for an incremental feature release,
but would have one distinguishing feature:

-   in a feature complete release,
    sys.version_info.feature_complete == True

Any subsequent maintenance and security fix only releases would also
have that flag set, and may informally be referred to as "feature
complete releases". For release series definition purposes though, the
feature complete release is the first one that sets that flag to "True".

Proposed policy adjustment for provisional APIs

To help improve consistency in management of provisional APIs, this PEP
proposes that provisional APIs be subject to regular backwards
compatibility requirements following the feature complete release for a
given release series.

Other aspects of managing provisional APIs would remain as they are
today, so as long as an API remains in the provisional state, regular
backwards compatibility requirements would not apply to that API in
baseline and incremental feature releases.

This policy is expected to provide increased clarity to end users (as
even provisional APIs will become stable for that release series in the
feature complete release), with minimal practical downsides for standard
library maintainers, based on the following analysis of documented API
additions and changes in micro releases of CPython since 3.0.0:

-   21 3.x.1 version added/changed notes
-   30 3.x.2 version added/changed notes
-   18 3.x.3 version added/changed notes
-   11 3.x.4 version added/changed notes
-   1 3.x.5 version added/changed notes
-   0 3.x.6+ version added/changed notes

When post-baseline-release changes need to be made, the majority of them
occur within the first two maintenance releases, which have always
occurred within 12 months of the baseline release.

(Note: these counts are not solely for provisional APIs - they cover all
APIs where semantic changes were made after the baseline release that
were considered necessary to cover in the documentation. To avoid double
counting changes, the numbers exclude any change markers from the What's
New section)

Motivation

The motivation for change in this PEP is essentially the same as the
motivation for change in PEP 596: the current 18-24 month gap between
feature releases has a lot of undesirable consequences, especially for
the standard library (see PEP 596 for further articulation of the
details).

This PEP's concern with the specific proposal in PEP 596 is that it
doubles the number of actively supported Python branches, increasing the
complexity of compatibility testing matrices for the entire Python
community, increasing the number of binary Python wheels to be uploaded
to PyPI when not using the stable ABI, and just generally having a high
chance of inflicting a relatively high level of additional cost across
the entire Python ecosystem.

The view taken in this PEP is that there's an alternative approach that
provides most of the benefits of a faster feature release without
actually incurring the associated costs: we can split the current X.Y.0
"feature freeze" into two parts, such that the baseline X.Y.0 release
only imposes a "runtime compatibility freeze", and the full standard
library feature freeze is deferred until later in the release series
lifecycle.

Caveats and Limitations

This proposal does NOT retroactively apply to Python 3.8 - it is being
proposed for Python 3.9 and later releases only.

Actual release dates may be adjusted up to a month earlier or later at
the discretion of the release manager, based on release team
availability, and the timing of other events (e.g. PyCon US, or the
annual core development sprints). However, part of the goal of this
proposal is to provide a consistent annual cadence for both contributors
and end users, so adjustments ideally would be rare.

This PEP does not dictate a specific cadence for micro releases within a
release series - it just specifies the rough timelines for transitions
between the release series lifecycle phases (pre-alpha, alpha, beta,
feature releases, bug fixes, security fixes). The number of micro
releases within each phase is determined by the release manager for that
series based on how frequently they and the rest of the release team for
that series are prepared to undertake the associated work.

However, for the sake of the example timelines, the PEP assumes
quarterly micro releases (the cadence used for Python 3.6 and 3.7,
splitting the difference between the twice yearly cadence used for some
historical release series, and the monthly cadence planned for Python
3.8 and 3.9).

Design Discussion

Why this proposal over simply doing more frequent baseline feature releases?

The filesystem layout changes and other inherently incompatible changes
involved in a baseline feature release create additional work for large
sections of the wider Python community.

Decoupling those layout changes from the Python version numbering scheme
is also something that would in and of itself involve making backwards
incompatible changes, as well as adjusting community expectations around
which versions will install over the top of each other, and which can be
installed in parallel on a single system.

We also don't have a straightforward means to communicate to the
community variations in support periods like "Only support Python
version X.Y until X.Y+1 is out, but support X.Z until X.Z+2 is out".

So this PEP takes as its starting assumption that the vast majority of
Python users simply shouldn't need to care that we're changing our
release policy, and the only folks that should be affected are those
that are eagerly waiting for standard library improvements (and other
backwards compatible interpreter enhancements), and those that need to
manage mission critical applications in complex deployment environments.

Implications for Python library development

Many Python libraries (both open source and proprietary) currently adopt
the practice of testing solely against the latest micro release within
each feature release series that the project still supports.

The design assumption in this PEP is that this practice will continue to
be followed during the feature release phase of a release series, with
the expectation being that anyone choosing to adopt a new release series
before it is feature complete will closely track the incremental feature
releases.

Libraries that support a previous feature release series are unlikely to
adopt features added in an incremental feature release, and if they do
adopt such a feature, then any associated fallback compatibility
strategies should be implemented in such a way that they're also
effective on the earlier releases in that release series.

Implications for the proposed Scientific Python ecosystem support period

Based on discussions at SciPy 2019, a NEP is currently being drafted[2]
to define a common convention across the Scientific Python ecosystem for
dropping support for older Python versions.

While the exact formulation of that policy is still being discussed, the
initial proposal was very simple: support any Python feature release
published within the last 42 months.

For an 18-month feature release cadence, that works out to always
supporting at least the two most recent feature releases, and then
dropping support for all X.Y.z releases around 6 months after X.(Y+2).0
is released. This means there is a 6-month period roughly every other
year where the three most recent feature releases are supported.

For a 12-month release cadence, it would work out to always supporting
at least the three most recent feature releases, and then dropping
support for all X.Y.z releases around 6 months after X.(Y+3).0 is
released. This means that for half of each year, the four most recent
feature releases would be supported.

For a 24-month release cadence, a 42-month support cycle works out to
always supporting at least the most recent feature release, and then
dropping support for all X.Y.z feature releases around 18 months after
X.(Y+1).0 is released. This means there is a 6-month period every other
year where only one feature release is supported (and that period
overlaps with the pre-release testing period for the X.(Y+2).0 baseline
feature release).

Importantly for the proposal in this PEP, that support period would
abide by the recommendation that library developers maintain support for
the previous release series until the latest release series has attained
feature complete status: dropping support 18 months after the baseline
feature release will be roughly equivalent to dropping support 6 months
after the feature complete release, without needing to track exactly
which release marked the series as feature complete.

Implications for simple deployment environments

For the purposes of this PEP, a "simple" deployment environment is any
use case where it is straightforward to ensure that all target
environments are updated to a new Python micro version at the same time
(or at least in advance of the rollout of new higher level application
versions), and there isn't any requirement for older Python versions to
be able to reliably read pickle streams generated with the newer Python
version, such that any pre-release testing that occurs need only target
a single Python micro version.

The simplest such case would be scripting for personal use, where the
testing and target environments are the exact same environment.

Similarly simple environments would be containerised web services, where
the same Python container is used in the CI pipeline as is used on
deployment, and any application that bundles its own Python runtime,
rather than relying on a pre-existing Python deployment on the target
system.

For these use cases, this PEP shouldn't have any significant
implications - only a single micro version needs to be tested,
independently of whether that version is feature complete or not.

Implications for complex deployment environments

For the purposes of this PEP, "complex" deployment environments are use
cases which don't meet the "simple deployment" criterion above: new
application versions are combined with two or more distinct micro
versions within the same release series as part of the deployment
process, rather than always targeting exactly one micro version at a
time.

If the proposal in this PEP has the desired effect of reducing feature
delivery latency, then it can be expected that developers using a
release series that is not yet feature complete will actually make use
of the new features as they're made available. This then means that
testing against a newer incremental feature release becomes an even less
valid test of compatibility with the baseline feature release and older
incremental feature releases than testing against a newer maintenance
release is for older maintenance releases.

One option for handling such cases is to simply prohibit the use of new
Python versions until the series has reached "feature complete" status.
Such a policy is effectively already adopted by many organisations when
it comes to new feature release series, with acceptance into operational
environments occurring months or years after the original release. If
this policy is adopted, then such organisations could potentially still
adopt a new Python version every other year - it would just be based on
the availability of the feature complete releases, rather than the
baseline feature releases.

A less strict alternative to outright prohibition would be to make use
of the proposed PYTHONFEATURELIMIT setting to enable phased migrations
to new incremental feature releases:

-   initially roll out Python X.Y.0 with PYTHONFEATURELIMIT=X.Y.0 set in
    CI and on deployment
-   roll out Python X.Y.1 to CI, keeping the PYTHONFEATURELIMIT=X.Y.0
    setting
-   deploy Python X.Y.1 to production based on successful CI results
-   update deployment environments to set PYTHONFEATURELIMIT=X.Y.1
-   set PYTHONFEATURELIMIT=X.Y.1 in CI only after all deployment
    environments have been updated
-   repeat this process for each new release up to and including the
    feature complete release for the release series
-   once the series is feature complete, either continue with this same
    process for consistency's sake, or else stop updating
    PYTHONFEATURELIMIT and leave it at the feature complete version
    number

Duration of the feature additions period

This PEP proposes that feature additions be limited to 12 months after
the initial baseline feature release.

The primary motivation for that is specifically to sync up with the
Ubuntu LTS timing, such that the feature complete release for the Python
3.9.x series gets published in October 2021, ready for inclusion in the
Ubuntu 22.04 release. (other LTS Linux distributions like RHEL, SLES,
and Debian don't have a fixed publishing cadence, so they can more
easily tweak their LTS timing a bit to align with stable versions of
their inputs. Canonical deliberately haven't given themselves that
flexibility with their own release cycle).

The 12 month feature addition period then arises from splitting the time
from the 2019-10 release of Python 3.8.0 and a final Python 3.9.x
incremental feature release in 2021-10 evenly between pre-release
development and subsequent incremental feature releases.

This is an area where this PEP could adopt part of the proposal in PEP
596, by instead making that split ~9 months of pre-release development,
and ~15 months of incremental feature releases:

-   2019-11: 3.9.0a1
-   ... additional alpha releases as determined by the release manager
-   2020-03: 3.9.0b1
-   2020-04: 3.9.0b2
-   2020-05: 3.9.0b3 (final beta release that locks ABI compatibility)
-   2020-06: 3.9.0rc1
-   ... additional release candidates as determined by the release
    manager
-   2020-07: 3.9.0 (BFR)
-   2020-10: 3.9.1 (IFR)
-   2021-01: 3.9.2 (IFR)
-   2021-04: 3.9.3 (IFR)
-   2021-07: 3.9.4 (IFR)
-   2021-10: 3.9.5
-   2022-01: 3.9.6
-   2022-04: 3.9.7
-   2022-07: 3.9.8
-   2022-10: 3.9.9 (final regular maintenance release)
-   ... additional security fix only releases as needed
-   2025-10: 3.9.x branch closed

This approach would mean there were still always two or three active
branches, it's just that proportionally more time would be spent with a
branch in the "feature releases" phase, as compared to the "pre-alpha",
"pre-beta", and "pre-release" phases:

-   2019-04 -> 2019-10: 3.9.0 pre-alpha, 3.8.0 pre-release, 3.7.x
    maintenance
-   2019-10 -> 2020-03: 3.9.0 pre-beta, 3.8.x maintenance
-   2020-03 -> 2020-07: 3.10.0 pre-alpha, 3.9.0 pre-release, 3.8.x
    maintenance
-   2020-07 -> 2021-10: 3.10.0 pre-alpha, 3.9.x feature releases, 3.8.x
    maintenance
-   2021-10 -> 2022-03: 3.10.0 pre-beta, 3.9.x maintenance
-   2022-03 -> 2022-07: 3.11.0 pre-alpha, 3.10.0 pre-release, 3.9.x
    maintenance
-   2022-07 -> 2023-10: 3.11.0 pre-alpha, 3.10.x feature releases, 3.9.x
    maintenance
-   2023-10 -> 2024-03: 3.11.0 pre-beta, 3.10.x maintenance
-   2024-03 -> 2024-07: 3.12.0 pre-alpha, 3.11.0 pre-release, 3.10.x
    maintenance
-   ... etc

Duration of the unreleased pre-alpha period

In the baseline proposal in this PEP, the proposed timelines still
include periods where we go for 18 months without making a release from
the main git branch (e.g. 3.9.0b1 would branch off in 2020-05, and
3.10.0a1 wouldn't be published until 2021-11). They just allow for a
wider variety of changes to be backported to the most recent maintenance
branch for 12 of those months.

The variant of the proposal that moves the beta branch point earlier in
the release series lifecycle would increase that period of no direct
releases to 21 months - the only period where releases were made
directly from the main branch would be during the relatively short
window between the last incremental feature release of the previous
release series, and the beta branch point a few months later.

While alternating the annual cadence between "big foundational
enhancements" and "targeted low risk API usability improvements" is a
deliberate feature of this proposal, it still seems strange to wait that
long for feedback in the event that changes are made shortly after the
previous release series is branched.

An alternative way of handling this would be to start publishing alpha
releases for the next baseline feature release during the feature
addition period (similar to the way that PEP 596 proposes to starting
publishing Python 3.9.0 alpha releases during the Python 3.8.0 release
candidate period).

However, rather than setting specific timelines for that at a policy
level, it may make sense to leave that decision to individual release
managers, based on the specific changes that are being proposed for the
release they're managing.

Why not switch directly to full semantic versioning?

If this were a versioning design document for a new language, it would
use semantic versioning: the policies described above for baseline
feature releases would be applied to X.0.0 releases, the policies for
incremental feature releases would be applied to X.Y.0 releases, and the
policies for maintenance releases would be applied to X.Y.Z releases.

The problem for Python specifically is that all the policies and
properties for parallel installation support and ABI compatibility
definitions are currently associated with the first two fields of the
version number, and it has been that way for the better part of thirty
years.

As a result, it makes sense to split out the policy question of
introducing incremental feature releases in the first place from the
technical question of making the version numbering scheme better match
the semantics of the different release types.

If the proposal in this PEP were to be accepted by the Steering Council
for Python 3.9, then a better time to tackle that technical question
would be for the subsequent October 2022 baseline feature release, as
there are already inherent compatibility risks associated with the
choice of either "Python 4.0" (erroneous checks for the major version
being exactly 3 rather than 3 or greater), or "Python 3.10" (code
incorrectly assuming that the minor version will always contain exactly
one decimal digit)[3].

While the text of this PEP assumes that the release published in 2022
will be 3.10 (as the PEP author personally considers that the more
reasonable and most likely choice), there are complex pros and cons on
both sides of that decision, and this PEP does arguably add a potential
pro in favour of choosing the "Python 4.0" option (with the caveat that
we would also need to amend the affected installation layout and
compatibility markers to only consider the major version number, rather
than both the major and minor version).

If such a version numbering change were to be proposed and accepted,
then the example 3.10.x timeline given above would instead become the
following 4.x series timeline:

-   2021-11: 4.0.0a1
-   ... additional alpha releases as determined by the release manager
-   2022-05: 4.0.0b1
-   ... additional beta releases as determined by the release manager
-   2022-08: 4.0.0bX (final beta release that locks ABI compatibility)
-   2022-09: 4.0.0rc1
-   ... additional release candidates as determined by the release
    manager
-   2022-10: 4.0.0 (BFR)
-   2023-01: 4.1.0 (IFR)
-   2023-04: 4.2.0 (IFR)
-   2023-07: 4.3.0 (IFR)
-   2023-10: 4.4.0 (IFR)
-   2024-01: 4.4.1
-   2024-04: 4.4.2
-   2024-07: 4.4.3
-   2024-10: 4.4.4 (final regular maintenance release)
-   ... additional security fix only releases as needed
-   2027-10: 4.x branch closed

And the 5 year schedule forecast would look like:

-   2019-04 -> 2019-10: 3.9.0 pre-alpha, 3.8.0 pre-release, 3.7.x
    maintenance
-   2019-10 -> 2020-05: 3.9.0 pre-beta, 3.8.x maintenance
-   2020-05 -> 2020-10: 4.0.0 pre-alpha, 3.9.0 pre-release, 3.8.x
    maintenance
-   2020-10 -> 2021-10: 4.0.0 pre-alpha, 3.9.x feature releases, 3.8.x
    maintenance
-   2021-10 -> 2022-05: 4.0.0 pre-beta, 3.9.x maintenance
-   2022-05 -> 2022-10: 5.0.0 pre-alpha, 4.0.0 pre-release, 3.9.x
    maintenance
-   2022-10 -> 2023-10: 5.0.0 pre-alpha, 4.x.0 feature releases, 3.9.x
    maintenance
-   2023-10 -> 2024-05: 5.0.0 pre-beta, 4.x.y maintenance
-   2024-05 -> 2024-10: 6.0.0 pre-alpha, 5.0.0 pre-release, 4.x.y
    maintenance
-   ... etc

References

Copyright

This document has been placed in the public domain.

[1] Draft Extended Lifecycle for Python (ELPython) design concept
(https://github.com/elpython/elpython-meta/blob/master/README.md)

[2] NEP proposing a standard policy for dropping support of old Python
versions (https://github.com/numpy/numpy/pull/14086)

[3] Anthony Sottile created a pseudo "Python 3.10" to find and fix such
issues (https://github.com/asottile-archive/python3.10)