PEP: 263 Title: Defining Python Source Code Encodings Author: Marc-André
Lemburg <mal@lemburg.com>, Martin von Löwis <martin@v.loewis.de> Status:
Final Type: Standards Track Content-Type: text/x-rst Created:
06-Jun-2001 Python-Version: 2.3 Post-History:

Abstract

This PEP proposes to introduce a syntax to declare the encoding of a
Python source file. The encoding information is then used by the Python
parser to interpret the file using the given encoding. Most notably this
enhances the interpretation of Unicode literals in the source code and
makes it possible to write Unicode literals using e.g. UTF-8 directly in
an Unicode aware editor.

Problem

In Python 2.1, Unicode literals can only be written using the Latin-1
based encoding "unicode-escape". This makes the programming environment
rather unfriendly to Python users who live and work in non-Latin-1
locales such as many of the Asian countries. Programmers can write their
8-bit strings using the favorite encoding, but are bound to the
"unicode-escape" encoding for Unicode literals.

Proposed Solution

I propose to make the Python source code encoding both visible and
changeable on a per-source file basis by using a special comment at the
top of the file to declare the encoding.

To make Python aware of this encoding declaration a number of concept
changes are necessary with respect to the handling of Python source code
data.

Defining the Encoding

Python will default to ASCII as standard encoding if no other encoding
hints are given.

To define a source code encoding, a magic comment must be placed into
the source files either as first or second line in the file, such as:

    # coding=<encoding name>

or (using formats recognized by popular editors):

    #!/usr/bin/python
    # -*- coding: <encoding name> -*-

or:

    #!/usr/bin/python
    # vim: set fileencoding=<encoding name> :

More precisely, the first or second line must match the following
regular expression:

    ^[ \t\f]*#.*?coding[:=][ \t]*([-_.a-zA-Z0-9]+)

The first group of this expression is then interpreted as encoding name.
If the encoding is unknown to Python, an error is raised during
compilation. There must not be any Python statement on the line that
contains the encoding declaration. If the first line matches the second
line is ignored.

To aid with platforms such as Windows, which add Unicode BOM marks to
the beginning of Unicode files, the UTF-8 signature \xef\xbb\xbf will be
interpreted as 'utf-8' encoding as well (even if no magic encoding
comment is given).

If a source file uses both the UTF-8 BOM mark signature and a magic
encoding comment, the only allowed encoding for the comment is 'utf-8'.
Any other encoding will cause an error.

Examples

These are some examples to clarify the different styles for defining the
source code encoding at the top of a Python source file:

1.  With interpreter binary and using Emacs style file encoding comment:

        #!/usr/bin/python
        # -*- coding: latin-1 -*-
        import os, sys
        ...

        #!/usr/bin/python
        # -*- coding: iso-8859-15 -*-
        import os, sys
        ...

        #!/usr/bin/python
        # -*- coding: ascii -*-
        import os, sys
        ...

2.  Without interpreter line, using plain text:

        # This Python file uses the following encoding: utf-8
        import os, sys
        ...

3.  Text editors might have different ways of defining the file's
    encoding, e.g.:

        #!/usr/local/bin/python
        # coding: latin-1
        import os, sys
        ...

4.  Without encoding comment, Python's parser will assume ASCII text:

        #!/usr/local/bin/python
        import os, sys
        ...

5.  Encoding comments which don't work:

    1.  Missing "coding:" prefix:

            #!/usr/local/bin/python
            # latin-1
            import os, sys
            ...

    2.  Encoding comment not on line 1 or 2:

            #!/usr/local/bin/python
            #
            # -*- coding: latin-1 -*-
            import os, sys
            ...

    3.  Unsupported encoding:

            #!/usr/local/bin/python
            # -*- coding: utf-42 -*-
            import os, sys
            ...

Concepts

The PEP is based on the following concepts which would have to be
implemented to enable usage of such a magic comment:

1.  The complete Python source file should use a single encoding.
    Embedding of differently encoded data is not allowed and will result
    in a decoding error during compilation of the Python source code.

    Any encoding which allows processing the first two lines in the way
    indicated above is allowed as source code encoding, this includes
    ASCII compatible encodings as well as certain multi-byte encodings
    such as Shift_JIS. It does not include encodings which use two or
    more bytes for all characters like e.g. UTF-16. The reason for this
    is to keep the encoding detection algorithm in the tokenizer simple.

2.  Handling of escape sequences should continue to work as it does now,
    but with all possible source code encodings, that is standard string
    literals (both 8-bit and Unicode) are subject to escape sequence
    expansion while raw string literals only expand a very small subset
    of escape sequences.

3.  Python's tokenizer/compiler combo will need to be updated to work as
    follows:

    1.  read the file
    2.  decode it into Unicode assuming a fixed per-file encoding
    3.  convert it into a UTF-8 byte string
    4.  tokenize the UTF-8 content
    5.  compile it, creating Unicode objects from the given Unicode data
        and creating string objects from the Unicode literal data by
        first reencoding the UTF-8 data into 8-bit string data using the
        given file encoding

Note that Python identifiers are restricted to the ASCII subset of the
encoding, and thus need no further conversion after step 4.

Implementation

For backwards-compatibility with existing code which currently uses
non-ASCII in string literals without declaring an encoding, the
implementation will be introduced in two phases:

1.  Allow non-ASCII in string literals and comments, by internally
    treating a missing encoding declaration as a declaration of
    "iso-8859-1". This will cause arbitrary byte strings to correctly
    round-trip between step 2 and step 5 of the processing, and provide
    compatibility with Python 2.2 for Unicode literals that contain
    non-ASCII bytes.

    A warning will be issued if non-ASCII bytes are found in the input,
    once per improperly encoded input file.

2.  Remove the warning, and change the default encoding to "ascii".

The builtin compile() API will be enhanced to accept Unicode as input.
8-bit string input is subject to the standard procedure for encoding
detection as described above.

If a Unicode string with a coding declaration is passed to compile(), a
SyntaxError will be raised.

SUZUKI Hisao is working on a patch; see[1] for details. A patch
implementing only phase 1 is available at[2].

Phases

Implementation of steps 1 and 2 above were completed in 2.3, except for
changing the default encoding to "ascii".

The default encoding was set to "ascii" in version 2.5.

Scope

This PEP intends to provide an upgrade path from the current
(more-or-less) undefined source code encoding situation to a more robust
and portable definition.

References

History

-   1.10 and above: see CVS history
-   1.8: Added '.' to the coding RE.
-   1.7: Added warnings to phase 1 implementation. Replaced the Latin-1
    default encoding with the interpreter's default encoding. Added
    tweaks to compile().
-   1.4 - 1.6: Minor tweaks
-   1.3: Worked in comments by Martin v. Loewis: UTF-8 BOM mark
    detection, Emacs style magic comment, two phase approach to the
    implementation

Copyright

This document has been placed in the public domain.

[1] Phase 2 implementation: https://bugs.python.org/issue534304

[2] Phase 1 implementation: https://bugs.python.org/issue526840