PEP: 397 Title: Python launcher for Windows Author: Mark Hammond
<mhammond@skippinet.com.au>, Martin von Löwis <martin@v.loewis.de>
Status: Final Type: Standards Track Content-Type: text/x-rst Created:
15-Mar-2011 Python-Version: 3.3 Post-History: 21-Jul-2011, 17-May-2011,
15-Mar-2011 Resolution:
https://mail.python.org/pipermail/python-dev/2012-June/120505.html

Abstract

This PEP describes a Python launcher for the Windows platform. A Python
launcher is a single executable which uses a number of heuristics to
locate a Python executable and launch it with a specified command line.

Rationale

Windows provides "file associations" so an executable can be associated
with an extension, allowing for scripts to be executed directly in some
contexts (eg., double-clicking the file in Windows Explorer.) Until now,
a strategy of "last installed Python wins" has been used and while not
ideal, has generally been workable due to the conservative changes in
Python 2.x releases. As Python 3.x scripts are often syntactically
incompatible with Python 2.x scripts, a different strategy must be used
to allow files with a '.py' extension to use a different executable
based on the Python version the script targets. This will be done by
borrowing the existing practices of another operating system - scripts
will be able to nominate the version of Python they need by way of a
"shebang" line, as described below.

Unix-like operating systems (referred to simply as "Unix" in this PEP)
allow scripts to be executed as if they were executable images by
examining the script for a "shebang" line which specifies the actual
executable to be used to run the script. This is described in detail in
the evecve(2) man page[1] and while user documentation will be created
for this feature, for the purposes of this PEP that man page describes a
valid shebang line.

Additionally, these operating systems provide symbolic-links to Python
executables in well-known directories. For example, many systems will
have a link /usr/bin/python which references a particular version of
Python installed under the operating-system. These symbolic links allow
Python to be executed without regard for where Python it actually
installed on the machine (eg., without requiring the path where Python
is actually installed to be referenced in the shebang line or in the
PATH.) PEP 394 'The "python" command on Unix-Like Systems' describes
additional conventions for more fine-grained specification of a
particular Python version.

These 2 facilities combined allow for a portable and somewhat
predictable way of both starting Python interactively and for allowing
Python scripts to execute. This PEP describes an implementation of a
launcher which can offer the same benefits for Python on the Windows
platform and therefore allows the launcher to be the executable
associated with '.py' files to support multiple Python versions
concurrently.

While this PEP offers the ability to use a shebang line which should
work on both Windows and Unix, this is not the primary motivation for
this PEP - the primary motivation is to allow a specific version to be
specified without inventing new syntax or conventions to describe it.

Specification

This PEP specifies features of the launcher; a prototype implementation
is provided in[2] which will be distributed together with the Windows
installer of Python, but will also be available separately (but released
along with the Python installer). New features may be added to the
launcher as long as the features prescribed here continue to work.

Installation

The launcher comes in 2 versions - one which is a console program and
one which is a "windows" (ie., GUI) program. These 2 launchers
correspond to the 'python.exe' and 'pythonw.exe' executables which
currently ship with Python. The console launcher will be named 'py.exe'
and the Windows one named 'pyw.exe'. The "windows" (ie., GUI) version of
the launcher will attempt to locate and launch pythonw.exe even if a
virtual shebang line nominates simply "python" - in fact, the trailing
'w' notation is not supported in the virtual shebang line at all.

The launcher is installed into the Windows directory (see discussion
below) if installed by a privileged user. The stand-alone installer asks
for an alternative location of the installer, and adds that location to
the user's PATH.

The installation in the Windows directory is a 32-bit executable (see
discussion); the standalone installer may also offer to install 64-bit
versions of the launcher.

The launcher installation is registered in
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\CurrentVersion\SharedDLLs with a
reference counter. It contains a version resource matching the version
number of the pythonXY.dll with which it is distributed. Independent
installations will overwrite older version of the launcher with newer
versions. Stand-alone releases use a release level of 0x10 in FIELD3 of
the CPython release on which they are based.

Once installed, the "console" version of the launcher is associated with
.py files and the "windows" version associated with .pyw files.

The launcher is not tied to a specific version of Python - eg., a
launcher distributed with Python 3.3 should be capable of locating and
executing any Python 2.x and Python 3.x version. However, the launcher
binaries have a version resource that is the same as the version
resource in the Python binaries that they are released with.

Python Script Launching

The launcher is restricted to launching Python scripts. It is not
intended as a general-purpose script launcher or shebang processor.

The launcher supports the syntax of shebang lines as described in[3],
including all restrictions listed.

The launcher supports shebang lines referring to Python executables with
any of the (regex) prefixes "/usr/bin/", "/usr/local/bin" and
"/usr/bin/env *", as well as binaries specified without

For example, a shebang line of '#! /usr/bin/python' should work even
though there is unlikely to be an executable in the relative Windows
directory "\usr\bin". This means that many scripts can use a single
shebang line and be likely to work on both Unix and Windows without
modification.

The launcher will support fully-qualified paths to executables. While
this will make the script inherently non-portable, it is a feature
offered by Unix and would be useful for Windows users in some cases.

The launcher will be capable of supporting implementations other than
CPython, such as jython and IronPython, but given both the absence of
common links on Unix (such as "/usr/bin/jython") and the inability for
the launcher to automatically locate the installation location of these
implementations on Windows, the launcher will support this via
customization options. Scripts taking advantage of this will not be
portable (as these customization options must be set to reflect the
configuration of the machine on which the launcher is running) but this
ability is nonetheless considered worthwhile.

On Unix, the user can control which specific version of Python is used
by adjusting the links in /usr/bin to point to the desired version. As
the launcher on Windows will not use Windows links, customization
options (exposed via both environment variables and INI files) will be
used to override the semantics for determining what version of Python
will be used. For example, while a shebang line of "/usr/bin/python2"
will automatically locate a Python 2.x implementation, an environment
variable can override exactly which Python 2.x implementation will be
chosen. Similarly for "/usr/bin/python" and "/usr/bin/python3". This is
specified in detail later in this PEP.

Shebang line parsing

If the first command-line argument does not start with a dash ('-')
character, an attempt will be made to open that argument as a file and
parsed for a shebang line according to the rules in[4]:

    #! interpreter [optional-arg]

Once parsed, the command will be categorized according to the following
rules:

-   If the command starts with the definition of a customized command
    followed by a whitespace character (including a newline), the
    customized command will be used. See below for a description of
    customized commands.
-   The launcher will define a set of prefixes which are considered Unix
    compatible commands to launch Python, namely "/usr/bin/python",
    "/usr/local/bin/python", "/usr/bin/env python", and "python". If a
    command starts with one of these strings will be treated as a
    'virtual command' and the rules described in Python Version
    Qualifiers (below) will be used to locate the executable to use.
-   Otherwise the command is assumed to be directly ready to execute -
    ie. a fully-qualified path (or a reference to an executable on the
    PATH) optionally followed by arguments. The contents of the string
    will not be parsed - it will be passed directly to the Windows
    CreateProcess function after appending the name of the script and
    the launcher command-line arguments. This means that the rules used
    by CreateProcess will be used, including how relative path names and
    executable references without extensions are treated. Notably, the
    Windows command processor will not be used, so special rules used by
    the command processor (such as automatic appending of extensions
    other than '.exe', support for batch files, etc) will not be used.

The use of 'virtual' shebang lines is encouraged as this should allow
for portable shebang lines to be specified which work on multiple
operating systems and different installations of the same operating
system.

If the first argument can not be opened as a file or if no valid shebang
line can be found, the launcher will act as if a shebang line of
'#!python' was found - ie., a default Python interpreter will be located
and the arguments passed to that. However, if a valid shebang line is
found but the process specified by that line can not be started, the
default interpreter will not be started - the error to create the
specified child process will cause the launcher to display an
appropriate message and terminate with a specific exit code.

Configuration file

Two .ini files will be searched by the launcher - py.ini in the current
user's "application data" directory (i.e. the directory returned by
calling the Windows function SHGetFolderPath with CSIDL_LOCAL_APPDATA,
%USERPROFILE%\AppData\Local on Vista+,
%USERPROFILE%\Local Settings\Application Data on XP) and py.ini in the
same directory as the launcher. The same .ini files are used for both
the 'console' version of the launcher (i.e. py.exe) and for the
'windows' version (i.e. pyw.exe)

Customization specified in the "application directory" will have
precedence over the one next to the executable, so a user, who may not
have write access to the .ini file next to the launcher, can override
commands in that global .ini file)

Virtual commands in shebang lines

Virtual Commands are shebang lines which start with strings which would
be expected to work on Unix platforms - examples include
'/usr/bin/python', '/usr/bin/env python' and 'python'. Optionally, the
virtual command may be suffixed with a version qualifier (see below),
such as '/usr/bin/python2' or '/usr/bin/python3.2'. The command executed
is based on the rules described in Python Version Qualifiers below.

Customized Commands

The launcher will support the ability to define "Customized Commands" in
a Windows .ini file (ie, a file which can be parsed by the Windows
function GetPrivateProfileString). A section called '[commands]' can be
created with key names defining the virtual command and the value
specifying the actual command-line to be used for this virtual command.

For example, if an INI file has the contents:

    [commands]
    vpython=c:\bin\vpython.exe -foo

Then a shebang line of '#! vpython' in a script named 'doit.py' will
result in the launcher using the command-line
c:\bin\vpython.exe -foo doit.py

The precise details about the names, locations and search order of the
.ini files is in the launcher documentation[5]

Python Version Qualifiers

Some of the features described allow an optional Python version
qualifier to be used.

A version qualifier starts with a major version number and can
optionally be followed by a period ('.') and a minor version specifier.
If the minor qualifier is specified, it may optionally be followed by
"-32" to indicate the 32bit implementation of that version be used. Note
that no "-64" qualifier is necessary as this is the default
implementation (see below).

On 64bit Windows with both 32bit and 64bit implementations of the same
(major.minor) Python version installed, the 64bit version will always be
preferred. This will be true for both 32bit and 64bit implementations of
the launcher - a 32bit launcher will prefer to execute a 64bit Python
installation of the specified version if available. This is so the
behavior of the launcher can be predicted knowing only what versions are
installed on the PC and without regard to the order in which they were
installed (ie, without knowing whether a 32 or 64bit version of Python
and corresponding launcher was installed last). As noted above, an
optional "-32" suffix can be used on a version specifier to change this
behaviour.

If no version qualifiers are found in a command, the environment
variable PY_PYTHON can be set to specify the default version qualifier -
the default value is "2". Note this value could specify just a major
version (e.g. "2") or a major.minor qualifier (e.g. "2.6"), or even
major.minor-32.

If no minor version qualifiers are found, the environment variable
PY_PYTHON{major} (where {major} is the current major version qualifier
as determined above) can be set to specify the full version. If no such
option is found, the launcher will enumerate the installed Python
versions and use the latest minor release found for the major version,
which is likely, although not guaranteed, to be the most recently
installed version in that family.

In addition to environment variables, the same settings can be
configured in the .INI file used by the launcher. The section in the INI
file is called [defaults] and the key name will be the same as the
environment variables without the leading PY_ prefix (and note that the
key names in the INI file are case insensitive.) The contents of an
environment variable will override things specified in the INI file.

Command-line handling

Only the first command-line argument will be checked for a shebang line
and only if that argument does not start with a '-'.

If the only command-line argument is "-h" or "--help", the launcher will
print a small banner and command-line usage, then pass the argument to
the default Python. This will cause help for the launcher being printed
followed by help for Python itself. The output from the launcher will
clearly indicate the extended help information is coming from the
launcher and not Python.

As a concession to interactively launching Python, the launcher will
support the first command-line argument optionally being a dash ("-")
followed by a version qualifier, as described above, to nominate a
specific version be used. For example, while "py.exe" may locate and
launch the latest Python 2.x implementation installed, a command-line
such as "py.exe -3" could specify the latest Python 3.x implementation
be launched, while "py.exe -2.6-32" could specify a 32bit implementation
Python 2.6 be located and launched. If a Python 2.x implementation is
desired to be launched with the -3 flag, the command-line would need to
be similar to "py.exe -2 -3" (or the specific version of Python could
obviously be launched manually without use of this launcher.) Note that
this feature can not be used with shebang processing as the file scanned
for a shebang line and this argument must both be the first argument and
therefore are mutually exclusive.

All other arguments will be passed untouched to the child Python
process.

Process Launching

The launcher offers some conveniences for Python developers working
interactively - for example, starting the launcher with no command-line
arguments will launch the default Python with no command-line arguments.
Further, command-line arguments will be supported to allow a specific
Python version to be launched interactively - however, these
conveniences must not detract from the primary purpose of launching
scripts and must be easy to avoid if desired.

The launcher creates a subprocess to start the actual interpreter. See
Discussion below for the rationale.

Discussion

It may be surprising that the launcher is installed into the Windows
directory, and not the System32 directory. The reason is that the
System32 directory is not on the Path of a 32-bit process running on a
64-bit system. However, the Windows directory is always on the path.

The launcher that is installed into the Windows directory is a 32-bit
executable so that the 32-bit CPython installer can provide the same
binary for both 32-bit and 64-bit Windows installations.

Ideally, the launcher process would execute Python directly inside the
same process, primarily so the parent of the launcher process could
terminate the launcher and have the Python interpreter terminate. If the
launcher executes Python as a sub-process and the parent of the launcher
terminates the launcher, the Python process will be unaffected.

However, there are a number of practical problems associated with this
approach. Windows does not support the execv* family of Unix functions,
so this could only be done by the launcher dynamically loading the
Python DLL, but this would have a number of side-effects. The most
serious side effect of this is that the value of sys.executable would
refer to the launcher instead of the Python implementation. Many Python
scripts use the value of sys.executable to launch child processes, and
these scripts may fail to work as expected if the launcher is used.
Consider a "parent" script with a shebang line of '#! /usr/bin/python3'
which attempts to launch a child script (with no shebang) via
sys.executable - currently the child is launched using the exact same
version running the parent script. If sys.executable referred to the
launcher the child would be likely executed using a Python 2.x version
and would be likely to fail with a SyntaxError.

Another hurdle is the support for alternative Python implementations
using the "customized commands" feature described above, where loading
the command dynamically into a running executable is not possible.

The final hurdle is the rules above regarding 64bit and 32bit programs
-a 32bit launcher would be unable to load the 64bit version of Python
and vice-versa.

Given these considerations, the launcher will execute its command in a
child process, remaining alive while the child process is executing,
then terminate with the same exit code as returned by the child. To
address concerns regarding the termination of the launcher not killing
the child, the Win32 Job API will be used to arrange so that the child
process is automatically killed when the parent is terminated (although
children of that child process will continue as is the case now.) As
this Windows API is available in Windows XP and later, this launcher
will not work on Windows 2000 or earlier.

References

Copyright

This document has been placed in the public domain.

[1] http://linux.die.net/man/2/execve

[2] https://bitbucket.org/vinay.sajip/pylauncher

[3] http://linux.die.net/man/2/execve

[4] http://linux.die.net/man/2/execve

[5] https://bitbucket.org/vinay.sajip/pylauncher/src/tip/Doc/launcher.rst