PEP: 454 Title: Add a new tracemalloc module to trace Python memory
allocations Version: $Revision$ Last-Modified: $Date$ Author: Victor
Stinner <vstinner@python.org> BDFL-Delegate: Charles-François Natali
<cf.natali@gmail.com> Status: Final Type: Standards Track Content-Type:
text/x-rst Created: 03-Sep-2013 Python-Version: 3.4 Resolution:
https://mail.python.org/pipermail/python-dev/2013-November/130491.html

Abstract

This PEP proposes to add a new tracemalloc module to trace memory blocks
allocated by Python.

Rationale

Classic generic tools like Valgrind can get the C traceback where a
memory block was allocated. Using such tools to analyze Python memory
allocations does not help because most memory blocks are allocated in
the same C function, in PyMem_Malloc() for example. Moreover, Python has
an allocator for small objects called "pymalloc" which keeps free blocks
for efficiency. This is not well handled by these tools.

There are debug tools dedicated to the Python language like Heapy
Pympler and Meliae which lists all alive objects using the garbage
collector module (functions like gc.get_objects(), gc.get_referrers()
and gc.get_referents()), compute their size (ex: using sys.getsizeof())
and group objects by type. These tools provide a better estimation of
the memory usage of an application. They are useful when most memory
leaks are instances of the same type and this type is only instantiated
in a few functions. Problems arise when the object type is very common
like str or tuple, and it is hard to identify where these objects are
instantiated.

Finding reference cycles is also a difficult problem. There are
different tools to draw a diagram of all references. These tools cannot
be used on large applications with thousands of objects because the
diagram is too huge to be analyzed manually.

Proposal

Using the customized allocation API from PEP 445, it becomes easy to set
up a hook on Python memory allocators. A hook can inspect Python
internals to retrieve Python tracebacks. The idea of getting the current
traceback comes from the faulthandler module. The faulthandler dumps the
traceback of all Python threads on a crash, here is the idea is to get
the traceback of the current Python thread when a memory block is
allocated by Python.

This PEP proposes to add a new tracemalloc module, a debug tool to trace
memory blocks allocated by Python. The module provides the following
information:

-   Traceback where an object was allocated
-   Statistics on allocated memory blocks per filename and per line
    number: total size, number and average size of allocated memory
    blocks
-   Computed differences between two snapshots to detect memory leaks

The API of the tracemalloc module is similar to the API of the
faulthandler module: enable() / start(), disable() / stop() and
is_enabled() / is_tracing() functions, an environment variable
(PYTHONFAULTHANDLER and PYTHONTRACEMALLOC), and a -X command line option
(-X faulthandler and -X tracemalloc). See the documentation of the
faulthandler module.

The idea of tracing memory allocations is not new. It was first
implemented in the PySizer project in 2005. PySizer was implemented
differently: the traceback was stored in frame objects and some Python
types were linked the trace with the name of object type. PySizer patch
on CPython adds an overhead on performances and memory footprint, even
if the PySizer was not used. tracemalloc attaches a traceback to the
underlying layer, to memory blocks, and has no overhead when the module
is not tracing memory allocations.

The tracemalloc module has been written for CPython. Other
implementations of Python may not be able to provide it.

API

To trace most memory blocks allocated by Python, the module should be
started as early as possible by setting the PYTHONTRACEMALLOC
environment variable to 1, or by using -X tracemalloc command line
option. The tracemalloc.start() function can be called at runtime to
start tracing Python memory allocations.

By default, a trace of an allocated memory block only stores the most
recent frame (1 frame). To store 25 frames at startup: set the
PYTHONTRACEMALLOC environment variable to 25, or use the
-X tracemalloc=25 command line option. The set_traceback_limit()
function can be used at runtime to set the limit.

Functions

clear_traces() function:

  Clear traces of memory blocks allocated by Python.

  See also stop().

get_object_traceback(obj) function:

  Get the traceback where the Python object obj was allocated. Return a
  Traceback instance, or None if the tracemalloc module is not tracing
  memory allocations or did not trace the allocation of the object.

  See also gc.get_referrers() and sys.getsizeof() functions.

get_traceback_limit() function:

  Get the maximum number of frames stored in the traceback of a trace.

  The tracemalloc module must be tracing memory allocations to get the
  limit, otherwise an exception is raised.

  The limit is set by the start() function.

get_traced_memory() function:

  Get the current size and maximum size of memory blocks traced by the
  tracemalloc module as a tuple: (size: int, max_size: int).

get_tracemalloc_memory() function:

  Get the memory usage in bytes of the tracemalloc module used to store
  traces of memory blocks. Return an int.

is_tracing() function:

  True if the tracemalloc module is tracing Python memory allocations,
  False otherwise.

  See also start() and stop() functions.

start(nframe: int=1) function:

  Start tracing Python memory allocations: install hooks on Python
  memory allocators. Collected tracebacks of traces will be limited to
  nframe frames. By default, a trace of a memory block only stores the
  most recent frame: the limit is 1. nframe must be greater or equal to
  1.

  Storing more than 1 frame is only useful to compute statistics grouped
  by 'traceback' or to compute cumulative statistics: see the
  Snapshot.compare_to() and Snapshot.statistics() methods.

  Storing more frames increases the memory and CPU overhead of the
  tracemalloc module. Use the get_tracemalloc_memory() function to
  measure how much memory is used by the tracemalloc module.

  The PYTHONTRACEMALLOC environment variable (PYTHONTRACEMALLOC=NFRAME)
  and the -X tracemalloc=NFRAME command line option can be used to start
  tracing at startup.

  See also stop(), is_tracing() and get_traceback_limit() functions.

stop() function:

  Stop tracing Python memory allocations: uninstall hooks on Python
  memory allocators. Clear also traces of memory blocks allocated by
  Python

  Call take_snapshot() function to take a snapshot of traces before
  clearing them.

  See also start() and is_tracing() functions.

take_snapshot() function:

  Take a snapshot of traces of memory blocks allocated by Python. Return
  a new Snapshot instance.

  The snapshot does not include memory blocks allocated before the
  tracemalloc module started to trace memory allocations.

  Tracebacks of traces are limited to get_traceback_limit() frames. Use
  the nframe parameter of the start() function to store more frames.

  The tracemalloc module must be tracing memory allocations to take a
  snapshot, see the start() function.

  See also the get_object_traceback() function.

Filter

Filter(inclusive: bool, filename_pattern: str, lineno: int=None, all_frames: bool=False)
class:

  Filter on traces of memory blocks.

  See the fnmatch.fnmatch() function for the syntax of filename_pattern.
  The '.pyc' and '.pyo' file extensions are replaced with '.py'.

  Examples:

  -   Filter(True, subprocess.__file__) only includes traces of the
      subprocess module
  -   Filter(False, tracemalloc.__file__) excludes traces of the
      tracemalloc module
  -   Filter(False, "<unknown>") excludes empty tracebacks

inclusive attribute:

  If inclusive is True (include), only trace memory blocks allocated in
  a file with a name matching filename_pattern at line number lineno.

  If inclusive is False (exclude), ignore memory blocks allocated in a
  file with a name matching filename_pattern at line number lineno.

lineno attribute:

  Line number (int) of the filter. If lineno is None, the filter matches
  any line number.

filename_pattern attribute:

  Filename pattern of the filter (str).

all_frames attribute:

  If all_frames is True, all frames of the traceback are checked. If
  all_frames is False, only the most recent frame is checked.

  This attribute is ignored if the traceback limit is less than 2. See
  the get_traceback_limit() function and Snapshot.traceback_limit
  attribute.

Frame

Frame class:

  Frame of a traceback.

  The Traceback class is a sequence of Frame instances.

filename attribute:

  Filename (str).

lineno attribute:

  Line number (int).

Snapshot

Snapshot class:

  Snapshot of traces of memory blocks allocated by Python.

  The take_snapshot() function creates a snapshot instance.

compare_to(old_snapshot: Snapshot, group_by: str, cumulative: bool=False)
method:

  Compute the differences with an old snapshot. Get statistics as a
  sorted list of StatisticDiff instances grouped by group_by.

  See the statistics() method for group_by and cumulative parameters.

  The result is sorted from the biggest to the smallest by: absolute
  value of StatisticDiff.size_diff, StatisticDiff.size, absolute value
  of StatisticDiff.count_diff, Statistic.count and then by
  StatisticDiff.traceback.

dump(filename) method:

  Write the snapshot into a file.

  Use load() to reload the snapshot.

filter_traces(filters) method:

  Create a new Snapshot instance with a filtered traces sequence,
  filters is a list of Filter instances. If filters is an empty list,
  return a new Snapshot instance with a copy of the traces.

  All inclusive filters are applied at once, a trace is ignored if no
  inclusive filters match it. A trace is ignored if at least one
  exclusive filter matches it.

load(filename) classmethod:

  Load a snapshot from a file.

  See also dump().

statistics(group_by: str, cumulative: bool=False) method:

  Get statistics as a sorted list of Statistic instances grouped by
  group_by:

    group_by      description
    ------------- --------------------------
    'filename'    filename
    'lineno'      filename and line number
    'traceback'   traceback

  If cumulative is True, cumulate size and count of memory blocks of all
  frames of the traceback of a trace, not only the most recent frame.
  The cumulative mode can only be used with group_by equals to
  'filename' and 'lineno' and traceback_limit greater than 1.

  The result is sorted from the biggest to the smallest by:
  Statistic.size, Statistic.count and then by Statistic.traceback.

traceback_limit attribute:

  Maximum number of frames stored in the traceback of traces: result of
  the get_traceback_limit() when the snapshot was taken.

traces attribute:

  Traces of all memory blocks allocated by Python: sequence of Trace
  instances.

  The sequence has an undefined order. Use the Snapshot.statistics()
  method to get a sorted list of statistics.

Statistic

Statistic class:

  Statistic on memory allocations.

  Snapshot.statistics() returns a list of Statistic instances.

  See also the StatisticDiff class.

count attribute:

  Number of memory blocks (int).

size attribute:

  Total size of memory blocks in bytes (int).

traceback attribute:

  Traceback where the memory block was allocated, Traceback instance.

StatisticDiff

StatisticDiff class:

  Statistic difference on memory allocations between an old and a new
  Snapshot instance.

  Snapshot.compare_to() returns a list of StatisticDiff instances. See
  also the Statistic class.

count attribute:

  Number of memory blocks in the new snapshot (int): 0 if the memory
  blocks have been released in the new snapshot.

count_diff attribute:

  Difference of number of memory blocks between the old and the new
  snapshots (int): 0 if the memory blocks have been allocated in the new
  snapshot.

size attribute:

  Total size of memory blocks in bytes in the new snapshot (int): 0 if
  the memory blocks have been released in the new snapshot.

size_diff attribute:

  Difference of total size of memory blocks in bytes between the old and
  the new snapshots (int): 0 if the memory blocks have been allocated in
  the new snapshot.

traceback attribute:

  Traceback where the memory blocks were allocated, Traceback instance.

Trace

Trace class:

  Trace of a memory block.

  The Snapshot.traces attribute is a sequence of Trace instances.

size attribute:

  Size of the memory block in bytes (int).

traceback attribute:

  Traceback where the memory block was allocated, Traceback instance.

Traceback

Traceback class:

  Sequence of Frame instances sorted from the most recent frame to the
  oldest frame.

  A traceback contains at least 1 frame. If the tracemalloc module
  failed to get a frame, the filename "<unknown>" at line number 0 is
  used.

  When a snapshot is taken, tracebacks of traces are limited to
  get_traceback_limit() frames. See the take_snapshot() function.

  The Trace.traceback attribute is an instance of Traceback instance.

Rejected Alternatives

Log calls to the memory allocator

A different approach is to log calls to malloc(), realloc() and free()
functions. Calls can be logged into a file or send to another computer
through the network. Example of a log entry: name of the function, size
of the memory block, address of the memory block, Python traceback where
the allocation occurred, timestamp.

Logs cannot be used directly, getting the current status of the memory
requires to parse previous logs. For example, it is not possible to get
directly the traceback of a Python object, like
get_object_traceback(obj) does with traces.

Python uses objects with a very short lifetime and so makes an extensive
use of memory allocators. It has an allocator optimized for small
objects (less than 512 bytes) with a short lifetime. For example, the
Python test suites calls malloc(), realloc() or free() 270,000 times per
second in average. If the size of log entry is 32 bytes, logging
produces 8.2 MB per second or 29.0 GB per hour.

The alternative was rejected because it is less efficient and has less
features. Parsing logs in a different process or a different computer is
slower than maintaining traces on allocated memory blocks in the same
process.

Prior Work

-   Python Memory Validator (2005-2013): commercial Python memory
    validator developed by Software Verification. It uses the Python
    Reflection API.
-   PySizer: Google Summer of Code 2005 project by Nick Smallbone.
-   Heapy (2006-2013): part of the Guppy-PE project written by Sverker
    Nilsson.
-   Draft PEP: Support Tracking Low-Level Memory Usage in CPython (Brett
    Canon, 2006)
-   Muppy: project developed in 2008 by Robert Schuppenies.
-   asizeof: a pure Python module to estimate the size of objects by
    Jean Brouwers (2008).
-   Heapmonitor: It provides facilities to size individual objects and
    can track all objects of certain classes. It was developed in 2008
    by Ludwig Haehne.
-   Pympler (2008-2011): project based on asizeof, muppy and HeapMonitor
-   objgraph (2008-2012)
-   Dozer: WSGI Middleware version of the CherryPy memory leak debugger,
    written by Marius Gedminas (2008-2013)
-   Meliae: Python Memory Usage Analyzer developed by John A Meinel
    since 2009
-   gdb-heap: gdb script written in Python by Dave Malcolm (2010-2011)
    to analyze the usage of the heap memory
-   memory_profiler: written by Fabian Pedregosa (2011-2013)
-   caulk: written by Ben Timby in 2012

See also Pympler Related Work.

Links

tracemalloc:

-   #18874: Add a new tracemalloc module to trace Python memory
    allocations
-   pytracemalloc on PyPI

Copyright

This document has been placed in the public domain.



  Local Variables: mode: indented-text indent-tabs-mode: nil
  sentence-end-double-space: t fill-column: 70 coding: utf-8 End: