PEP: 306 Title: How to Change Python's Grammar Version: $Revision$
Last-Modified: $Date$ Author: Michael Hudson <mwh@python.net>, Jack
Diederich <jackdied@gmail.com>, Alyssa Coghlan <ncoghlan@gmail.com>,
Benjamin Peterson <benjamin@python.org> Status: Withdrawn Type:
Informational Content-Type: text/x-rst Created: 29-Jan-2003
Post-History: 30-Jan-2003

Note

This PEP has been moved to the Python dev guide[1].

Abstract

There's more to changing Python's grammar than editing Grammar/Grammar
and Python/compile.c. This PEP aims to be a checklist of places that
must also be fixed.

It is probably incomplete. If you see omissions, just add them if you
can -- you are not going to offend the author's sense of ownership.
Otherwise submit a bug or patch and assign it to mwh.

This PEP is not intended to be an instruction manual on Python grammar
hacking, for several reasons.

Rationale

People are getting this wrong all the time; it took well over a year
before someone noticed[2] that adding the floor division operator (//)
broke the parser module.

Checklist

-   Grammar/Grammar: OK, you'd probably worked this one out :)
-   Parser/Python.asdl may need changes to match the Grammar. Run make
    to regenerate Include/Python-ast.h and Python/Python-ast.c.
-   Python/ast.c will need changes to create the AST objects involved
    with the Grammar change. Lib/compiler/ast.py will need matching
    changes to the pure-python AST objects.
-   Parser/pgen needs to be rerun to regenerate Include/graminit.h and
    Python/graminit.c. (make should handle this for you.)
-   Python/symbtable.c: This handles the symbol collection pass that
    happens immediately before the compilation pass.
-   Python/compile.c: You will need to create or modify the compiler_*
    functions to generate opcodes for your productions.
-   You may need to regenerate Lib/symbol.py and/or Lib/token.py and/or
    Lib/keyword.py.
-   The parser module. Add some of your new syntax to test_parser, bang
    on Modules/parsermodule.c until it passes.
-   Add some usage of your new syntax to test_grammar.py.
-   The compiler package. A good test is to compile the standard library
    and test suite with the compiler package and then check it runs.
    Note that this only needs to be done in Python 2.x.
-   If you've gone so far as to change the token structure of Python,
    then the Lib/tokenizer.py library module will need to be changed.
-   Certain changes may require tweaks to the library module pyclbr.
-   Documentation must be written!
-   After everything's been checked in, you're likely to see a new
    change to Python/Python-ast.c. This is because this (generated) file
    contains the SVN version of the source from which it was generated.
    There's no way to avoid this; you just have to submit this file
    separately.

References

Copyright

This document has been placed in the public domain.

[1] CPython Developer's Guide: Changing CPython's Grammar
https://devguide.python.org/grammar/

[2] SF Bug #676521, parser module validation failure
https://bugs.python.org/issue676521