PEP: 389 Title: argparse - New Command Line Parsing Module Version:
$Revision$ Last-Modified: $Date$ Author: Steven Bethard
<steven.bethard@gmail.com> Status: Final Type: Standards Track
Content-Type: text/x-rst Created: 25-Sep-2009 Python-Version: 2.7, 3.2
Post-History: 27-Sep-2009, 24-Oct-2009

Acceptance

This PEP was approved by Guido on python-dev on February 21, 2010[1].

Abstract

This PEP proposes inclusion of the argparse[2] module in the Python
standard library in Python 2.7 and 3.2.

Motivation

The argparse module is a command line parsing library which provides
more functionality than the existing command line parsing modules in the
standard library, getopt[3] and optparse[4]. It includes support for
positional arguments (not just options), subcommands, required options,
options syntaxes like "/f" and "+rgb", zero-or-more and one-or-more
style arguments, and many other features the other two lack.

The argparse module is also already a popular third-party replacement
for these modules. It is used in projects like IPython (the Scipy Python
shell)[5], is included in Debian testing and unstable[6], and since 2007
has had various requests for its inclusion in the standard
library[7][8][9]. This popularity suggests it may be a valuable addition
to the Python libraries.

Why aren't getopt and optparse enough?

One argument against adding argparse is that there are "already two
different option parsing modules in the standard library"[10]. The
following is a list of features provided by argparse but not present in
getopt or optparse:

-   While it is true there are two option parsing libraries, there are
    no full command line parsing libraries -- both getopt and optparse
    support only options and have no support for positional arguments.
    The argparse module handles both, and as a result, is able to
    generate better help messages, avoiding redundancies like the usage=
    string usually required by optparse.
-   The argparse module values practicality over purity. Thus, argparse
    allows required options and customization of which characters are
    used to identify options, while optparse explicitly states "the
    phrase 'required option' is self-contradictory" and that the option
    syntaxes -pf, -file, +f, +rgb, /f and /file "are not supported by
    optparse, and they never will be".
-   The argparse module allows options to accept a variable number of
    arguments using nargs='?', nargs='*' or nargs='+'. The optparse
    module provides an untested recipe for some part of this
    functionality[11] but admits that "things get hairy when you want an
    option to take a variable number of arguments."
-   The argparse module supports subcommands, where a main command line
    parser dispatches to other command line parsers depending on the
    command line arguments. This is a common pattern in command line
    interfaces, e.g. svn co and svn up.

Why isn't the functionality just being added to optparse?

Clearly all the above features offer improvements over what is available
through optparse. A reasonable question then is why these features are
not simply provided as patches to optparse, instead of introducing an
entirely new module. In fact, the original development of argparse
intended to do just that, but because of various fairly constraining
design decisions of optparse, this wasn't really possible. Some of the
problems included:

-   The optparse module exposes the internals of its parsing algorithm.
    In particular, parser.largs and parser.rargs are guaranteed to be
    available to callbacks[12]. This makes it extremely difficult to
    improve the parsing algorithm as was necessary in argparse for
    proper handling of positional arguments and variable length
    arguments. For example, nargs='+' in argparse is matched using
    regular expressions and thus has no notion of things like
    parser.largs.

-   The optparse extension APIs are extremely complex. For example, just
    to use a simple custom string-to-object conversion function, you
    have to subclass Option, hack class attributes, and then specify
    your custom option type to the parser, like this:

        class MyOption(Option):
            TYPES = Option.TYPES + ("mytype",)
            TYPE_CHECKER = copy(Option.TYPE_CHECKER)
            TYPE_CHECKER["mytype"] = check_mytype
        parser = optparse.OptionParser(option_class=MyOption)
        parser.add_option("-m", type="mytype")

    For comparison, argparse simply allows conversion functions to be
    used as type= arguments directly, e.g.:

        parser = argparse.ArgumentParser()
        parser.add_option("-m", type=check_mytype)

    But given the baroque customization APIs of optparse, it is unclear
    how such a feature should interact with those APIs, and it is quite
    possible that introducing the simple argparse API would break
    existing custom Option code.

-   Both optparse and argparse parse command line arguments and assign
    them as attributes to an object returned by parse_args. However, the
    optparse module guarantees that the take_action method of custom
    actions will always be passed a values object which provides an
    ensure_value method[13], while the argparse module allows attributes
    to be assigned to any object, e.g.:

        foo_object = ...
        parser.parse_args(namespace=foo_object)
        foo_object.some_attribute_parsed_from_command_line

    Modifying optparse to allow any object to be passed in would be
    difficult because simply passing the foo_object around instead of a
    Values instance will break existing custom actions that depend on
    the ensure_value method.

Because of issues like these, which made it unreasonably difficult for
argparse to stay compatible with the optparse APIs, argparse was
developed as an independent module. Given these issues, merging all the
argparse features into optparse with no backwards incompatibilities
seems unlikely.

Deprecation of optparse

Because all of optparse's features are available in argparse, the
optparse module will be deprecated. However, because of the widespread
use of optparse, the deprecation strategy contains only documentation
changes and warnings that will not be visible by default:

-   Python 2.7+ and 3.2+ -- The following note will be added to the
    optparse documentation:

      The optparse module is deprecated and will not be developed
      further; development will continue with the argparse module.

-   Python 2.7+ -- If the Python 3 compatibility flag, -3, is provided
    at the command line, then importing optparse will issue a
    DeprecationWarning. Otherwise no warnings will be issued.

-   Python 3.2+ -- Importing optparse will issue a
    PendingDeprecationWarning, which is not displayed by default.

Note that no removal date is proposed for optparse.

Updates to getopt documentation

The getopt module will not be deprecated. However, its documentation
will be updated to point to argparse in a couple of places. At the top
of the module, the following note will be added:

  The getopt module is a parser for command line options whose API is
  designed to be familiar to users of the C getopt function. Users who
  are unfamiliar with the C getopt function or who would like to write
  less code and get better help and error messages should consider using
  the argparse module instead.

Additionally, after the final getopt example, the following note will be
added:

  Note that an equivalent command line interface could be produced with
  less code by using the argparse module:

      import argparse

      if __name__ == '__main__':
          parser = argparse.ArgumentParser()
          parser.add_argument('-o', '--output')
          parser.add_argument('-v', dest='verbose', action='store_true')
          args = parser.parse_args()
          # ... do something with args.output ...
          # ... do something with args.verbose ..

Deferred: string formatting

The argparse module supports Python from 2.3 up through 3.2 and as a
result relies on traditional %(foo)s style string formatting. It has
been suggested that it might be better to use the new style {foo} string
formatting[14]. There was some discussion about how best to do this for
modules in the standard library[15] and several people are developing
functions for automatically converting %-formatting to
{}-formatting[16][17]. When one of these is added to the standard
library, argparse will use them to support both formatting styles.

Rejected: getopt compatibility methods

Previously, when this PEP was suggesting the deprecation of getopt as
well as optparse, there was some talk of adding a method like:

    ArgumentParser.add_getopt_arguments(options[, long_options])

However, this method will not be added for a number of reasons:

-   The getopt module is not being deprecated, so there is less need.
-   This method would not actually ease the transition for any getopt
    users who were already maintaining usage messages, because the API
    above gives no way of adding help messages to the arguments.
-   Some users of getopt consider it very important that only a single
    function call is necessary. The API above does not satisfy this
    requirement because both ArgumentParser() and parse_args() must also
    be called.

Out of Scope: Various Feature Requests

Several feature requests for argparse were made in the discussion of
this PEP:

-   Support argument defaults from environment variables
-   Support argument defaults from configuration files
-   Support "foo --help subcommand" in addition to the currently
    supported "foo subcommand --help"

These are all reasonable feature requests for the argparse module, but
are out of the scope of this PEP, and have been redirected to the
argparse issue tracker.

Discussion: sys.stderr and sys.exit

There were some concerns that argparse by default always writes to
sys.stderr and always calls sys.exit when invalid arguments are
provided. This is the desired behavior for the vast majority of argparse
use cases which revolve around simple command line interfaces. However,
in some cases, it may be desirable to keep argparse from exiting, or to
have it write its messages to something other than sys.stderr. These use
cases can be supported by subclassing ArgumentParser and overriding the
exit or _print_message methods. The latter is an undocumented
implementation detail, but could be officially exposed if this turns out
to be a common need.

References

Copyright

This document has been placed in the public domain.



  Local Variables: mode: indented-text indent-tabs-mode: nil
  sentence-end-double-space: t fill-column: 70 coding: utf-8 End:

[1] Guido's approval
(https://mail.python.org/pipermail/python-dev/2010-February/097839.html)

[2] argparse (http://code.google.com/p/argparse/)

[3] getopt (http://docs.python.org/library/getopt.html)

[4] optparse (http://docs.python.org/library/optparse.html)

[5] argparse in IPython
(http://mail.scipy.org/pipermail/ipython-dev/2009-April/005102.html)

[6] argparse in Debian
(http://packages.debian.org/search?keywords=argparse)

[7] 2007-01-03 request for argparse in the standard library
(https://mail.python.org/pipermail/python-list/2007-January/472276.html)

[8] 2009-06-09 request for argparse in the standard library
(http://bugs.python.org/issue6247)

[9] 2009-09-10 request for argparse in the standard library
(https://mail.python.org/pipermail/stdlib-sig/2009-September/000342.html)

[10] Fredrik Lundh response to
(https://mail.python.org/pipermail/python-list/2007-January/1086892.html)

[11] optparse variable args
(http://docs.python.org/library/optparse.html#callback-example-6-variable-arguments)

[12] parser.largs and parser.rargs
(http://docs.python.org/library/optparse.html#how-callbacks-are-called)

[13] take_action values argument
(http://docs.python.org/library/optparse.html#adding-new-actions)

[14] use {}-formatting instead of %-formatting
(http://bugs.python.org/msg89279)

[15] transitioning from % to {} formatting
(https://mail.python.org/pipermail/python-dev/2009-September/092326.html)

[16] Vinay Sajip's %-to-{} converter (http://gist.github.com/200936)

[17] Benjamin Peterson's %-to-{} converter
(http://bazaar.launchpad.net/~gutworth/+junk/mod2format/files)