API

The module version can be read from pyperf.VERSION (tuple of int) or pyperf.__version__ (str).

See API examples.

Functions

add_runs(filename: str, result)

Append a Benchmark or BenchmarkSuite to an existing benchmark suite file, or create a new file.

If the file already exists, adds runs to existing benchmarks.

See BenchmarkSuite.add_runs() method.

format_metadata(name: str, value)

Format a metadata value. The formatter depends on name.

See Metadata.

perf_counter()

Deprecated alias to time.perf_counter(): use time.perf_counter() directly.

Deprecated since version 2.0.

python_implementation()

Name of the Python implementation in lower case.

Examples:

  • cpython

  • ironpython

  • jython

  • pypy

Use sys.implementation.name or platform.python_implementation().

See also the PEP 421.

python_has_jit()

Return True if Python has a Just In Time compiler (JIT).

For example, return True for PyPy but False for CPython.

Run class

class Run(values: Sequence[float], warmups: Sequence[float] = None, metadata: dict = None, collect_metadata=True)

A benchmark run result is made of multiple values.

values must be a sequence of numbers (integer or float) greater than zero. Values must be normalized per loop iteration. Usually, values is a list of number of seconds.

warmups is an optional sequence of (loops: int, value) tuples where value must be a number (integer or float) greater than or equal to zero. Warmup values must be normalized per loop iteration.

values and/or warmups must be a non-empty sequence. If values is empty, the run is a calibration run.

Values must not be equal to zero. If a value is zero, use more loop iterations: see Runs, values, warmups, outer and inner loops.

metadata are metadata of the run, see Metadata. Important metadata:

  • name (mandatory, non-empty str): benchmark name

  • loops (int >= 1): number of outer-loops

  • inner_loops (int >= 1): number of inner-loops

  • unit (str): unit of values: 'second', 'byte' or 'integer'

Set collect_metadata to false to not collect system metadata.

Methods:

get_metadata() → dict

Get run metadata.

The format_metadata() function can be used to format values.

See Metadata.

get_loops() → int

Get the number of outer loop iterations from metadata.

Return 1 if metadata have no 'loops' entry.

Added in version 1.3.

get_inner_loops() → int

Get the number of inner loop iterations from metadata.

Return 1 if metadata have no 'inner_loops' entry.

Added in version 1.3.

get_total_loops() → int

Get the total number of loops of the benchmark run: get_loops() x get_inner_loops().

Attributes:

values

Benchmark run values (tuple of numbers).

warmups

Benchmark warmup values (tuple of numbers).

Benchmark class

class Benchmark(runs)

A benchmark is made of multiple Run objects.

runs must be non-empty sequence of Run objects. Runs must have a name metadata (all runs must have the same name).

Methods:

add_run(run: Run)

Add a benchmark run: run must a Run object.

The new run must be compatible with existing runs, the following metadata must be the same (same value or no value for all runs):

  • aslr

  • cpu_count

  • cpu_model_name

  • hostname

  • inner_loops

  • name

  • platform

  • python_executable

  • python_implementation

  • python_version

  • unit

add_runs(bench: Benchmark)

Add runs of the benchmark bench.

See BenchmarkSuite.add_runs() method and add_runs() function.

dump(file, compact=True, replace=False)

Dump the benchmark as JSON into file.

file can be a filename, or a file object open for write.

If file is a filename ending with .gz, the file is compressed by gzip.

If file is a filename and replace is false, the function fails if the file already exists.

If compact is true, generate compact file. Otherwise, indent JSON.

See the pyperf JSON format.

format_value(value) → str

Format a value including the unit.

format_values(values) → str

Format values including the unit.

get_dates() → (datetime.datetime, datetime.datetime) or None

Get the start date of the first run and the end date of the last run.

Return a (start, end) tuple where start and end are datetime.datetime objects if a least one run has a date metadata.

Return None if no run has the date metadata.

get_metadata() → dict

Get metadata common to all runs.

The format_metadata() function can be used to format values.

See Metadata.

get_name() → str

Get the benchmark name (str).

get_nrun() → int

Get the number of runs.

get_nvalue() → int

Get the total number of values.

get_nwarmup() → int or float

Get the number of warmup values per run.

Return an int if all runs use the same number of warmups, or return the average as a float.

get_runs() → List[Run]

Get the list of Run objects.

get_values()

Get values of all runs.

get_total_duration() → float

Get the total duration of the benchmark in seconds.

Use the duration metadata of runs, or compute the sum of their raw values including warmup values.

get_loops() → int or float

Get the number of outer loop iterations of runs.

Return an int if all runs have the same number of outer loops, return the average as a float otherwise.

Added in version 1.3.

get_inner_loops() → int or float

Get the number of inner loop iterations of runs.

Return an int if all runs have the same number of outer loops, return the average as a float otherwise.

Added in version 1.3.

get_total_loops() → int or float

Get the total number of loops per value (outer-loops x inner-loops).

Return an int if all runs have the same number of loops, return the average as a float otherwise.

get_unit() → str

Get the unit of values:

  • 'byte': File size in bytes

  • 'integer': Integer number

  • 'second': Duration in seconds

classmethod load(file) → Benchmark

Load a benchmark from a JSON file which was created by dump().

file can be a filename, '-' string to load from sys.stdin, or a file object open to read.

Raise an exception if the file contains more than one benchmark.

See the pyperf JSON format.

classmethod loads(string) → Benchmark

Load a benchmark from a JSON string.

Raise an exception if JSON contains more than one benchmark.

See the pyperf JSON format.

mean()

Compute the arithmetic mean of get_values().

The mean is greater than zero: add_run() raises an error if a value is equal to zero.

Raise an exception if the benchmark has no values.

median()

Compute the median of get_values().

The median is greater than zero: add_run() raises an error if a value is equal to zero.

Raise an exception if the benchmark has no values.

percentile(p)

Compute the p-th percentile of get_values().

p must be in the range [0; 100]:

  • p=0 computes the minimum

  • p=25 computes Q1

  • p=50 computes the median (see also the median() method)

  • p=75 computes Q3

  • p=100 computes the maximum

stdev()

Compute the standard deviation of get_values().

Raise an exception if the benchmark has less than 2 values.

median_abs_dev()

Compute the median absolute deviation (MAD) of get_values().

Raise an exception if the benchmark has no values.

update_metadata(metadata: dict)

Update metadata of all runs of the benchmark.

If the inner_loops metadata is already set and its value is modified, an exception is raised.

See Metadata.

BenchmarkSuite class

class BenchmarkSuite(benchmarks, filename=None)

A benchmark suite is made of Benchmark objects.

benchmarks must be a non-empty sequence of Benchmark objects. filename is the name of the file from which the suite was loaded.

Methods:

add_benchmark(benchmark: Benchmark)

Add a Benchmark object.

A suite cannot contain two benchmarks with the same name, because the name is used as a unique key: see the get_benchmark() method.

add_runs(bench: Benchmark or BenchmarkSuite)

Add runs of benchmarks.

bench can be a Benchmark or a BenchmarkSuite.

See Benchmark.add_runs() method and add_runs() function.

dump(file, compact=True, replace=False)

Dump the benchmark suite as JSON into file.

file can be a filename, or a file object open for write.

If file is a filename ending with .gz, the file is compressed by gzip.

If file is a filename and replace is false, the function fails if the file already exists.

If compact is true, generate compact file. Otherwise, indent JSON.

See the pyperf JSON format.

get_benchmark(name: str) → Benchmark

Get the benchmark called name.

name must be non-empty.

Raise KeyError if there is no benchmark called name.

get_benchmark_names() → List[str]

Get the list of benchmark names.

get_benchmarks() → List[Benchmark]

Get the list of benchmarks.

get_dates() → (datetime.datetime, datetime.datetime) or None

Get the start date of the first benchmark and end date of the last benchmark.

Return a (start, end) tuple where start and end are datetime.datetime objects if a least one benchmark has dates.

Return None if no benchmark has dates.

get_metadata() → dict

Get metadata common to all benchmarks (common to all runs of all benchmarks).

The format_metadata() function can be used to format values.

See the Benchmark.get_metadata() method and Metadata.

get_total_duration() → float

Get the total duration of all benchmarks in seconds.

See the Benchmark.get_total_duration() method.

__iter__()

Iterate on benchmarks.

__len__() → int

Get the number of benchmarks.

classmethod load(file)

Load a benchmark suite from a JSON file which was created by dump().

file can be a filename, '-' string to load from sys.stdin, or a file object open to read.

See the pyperf JSON format.

classmethod loads(string) → Benchmark

Load a benchmark suite from a JSON string.

See the pyperf JSON format.

Attributes:

filename

Name of the file from which the benchmark suite was loaded. It can be None.

Runner class

class Runner(values=3, warmups=1, processes=20, loops=0, min_time=0.1, metadata=None, show_name=True, program_args=None, add_cmdline_args=None)

Tool to run a benchmark in text mode.

Spawn processes worker processes to run the benchmark.

metadata is passed to the Run constructor.

values, warmups and processes are the default number of values, warmup values and processes. These values can be changed with command line options. See Runner CLI for command line options.

program_args is a list of strings passed to Python on the command line to run the program. By default, (sys.argv[0],) is used. For example, python3 -m pyperf timeit sets program_args to ('-m', 'pyperf', 'timeit').

add_cmdline_args is an optional callback used to add command line arguments to the command line of worker processes. The callback is called with add_cmdline_args(cmd, args) where cmd is the command line (list) which must be modified in place and args is the args attribute of the runner.

If show_name is true, displays the benchmark name.

If isolated CPUs are detected, the CPU affinity is automatically set to these isolated CPUs. See CPU pinning and CPU isolation.

Methods to run benchmarks:

Only once instance of Runner must be created. Use the same instance to run all benchmarks.

Methods:

bench_func(name, func, \*args, inner_loops=None, metadata=None)

Benchmark the function func(*args).

name is the benchmark name, it must be unique in the same script.

The inner_loops parameter is used to normalize timing per loop iteration.

The design of bench_func() has a non negligible overhead on microbenchmarks: each loop iteration calls func(*args) but Python function calls are expensive. The timeit() and bench_time_func() methods are recommended if func(*args) takes less than 1 millisecond (0.001 second).

To call func() with keyword arguments, use functools.partial.

Return a Benchmark instance.

See the bench_func() example.

bench_async_func(name, func, \*args, inner_loops=None, metadata=None, loop_factory=None)

Benchmark the function await func(*args) in asyncio event loop.

name is the benchmark name, it must be unique in the same script.

The inner_loops parameter is used to normalize timing per loop iteration.

The loop_factory parameter, if specified, will be used to create the event loop used by the benchmark.

To call func() with keyword arguments, use functools.partial.

Return a Benchmark instance.

See the bench_async_func() example.

timeit(name, stmt=None, setup='pass', teardown='pass', inner_loops=None, duplicate=None, metadata=None, globals=None)

Run a benchmark on timeit.Timer(stmt, setup, globals=globals).

name is the benchmark name, it must be unique in the same script.

stmt is a Python statement. It can be a non-empty string or a non-empty sequence of strings.

setup is a Python statement used to setup the benchmark: it is executed before computing each benchmark value. It can be a string or a sequence of strings.

teardown is a Python statement used to teardown the benchmark: it is executed after computing each benchmark value. It can be a string or a sequence of strings.

Parameters:

  • inner_loops: Number of inner-loops. Can be used when stmt manually duplicates the same expression inner_loops times.

  • duplicate: Duplicate the stmt statement duplicate times to reduce the cost of the outer loop.

  • metadata: Metadata of this benchmark, added to the runner metadata.

  • globals: Namespace used to run setup, teardown and stmt. By default, an empty namespace is created. It can be used to pass variables.

Runner.timeit(stmt) can be used to use the statement as the benchmark name.

See the timeit() example.

Changed in version 1.6.0: Add optional teardown parameter. The stmt parameter is now optional.

bench_command(name, command)

Benchmark the execution time of a command using time.perf_counter() timer. Measure the wall-time, not CPU time.

command must be a sequence of arguments, the first argument must be the program.

Basically, the function measures the timing of Popen(command).wait(), but tries to reduce the benchmark overhead.

Standard streams (stdin, stdout and stderr) are redirected to /dev/null (or NUL on Windows).

Use --inherit-environ and --no-locale command line options to control environment variables.

If the resource.getrusage() function is available, measure also the maximum RSS memory and stores it in command_max_rss metadata.

See the bench_command() example.

Changed in version 1.1: Measure the maximum RSS memory (if available).

bench_time_func(name, time_func, \*args, inner_loops=None, metadata=None)

Benchmark time_func(loops, *args). The time_func function must return raw timings: the total elapsed time of all loops. Runner will divide raw timings by loops x inner_loops (loops and inner_loops parameters).

time.perf_counter() should be used to measure the elapsed time.

name is the benchmark name, it must be unique in the same script.

To call time_func() with keyword arguments, use functools.partial.

Return a Benchmark instance.

See the bench_time_func() example.

parse_args(args=None)

Parse command line arguments using argparser and put the result into the args attribute.

If args is set, the method must only be called once.

Return the args attribute.

Attributes:

args

Namespace of arguments: result of the parse_args() method, None before parse_args() is called.

argparser

An argparse.ArgumentParser object used to parse command line options.

metadata

Benchmark metadata (dict).

Metadata

The Run class collects metadata by default.

Benchmark:

  • date (str): date when the benchmark run started, formatted as ISO 8601

  • duration (int or float >= 0): total duration of the benchmark run in seconds (float)

  • name (non-empty str): benchmark name

  • loops (int >= 1): number of outer-loops per value (int)

  • inner_loops (int >= 1): number of inner-loops of the benchmark (int)

  • timer: Implementation of time.perf_counter(), and also resolution if available

  • tags: (list of str, optional): A list of tags associated with the benchmark. If provided, the results output will be aggreggated by each tag.

Python metadata:

  • python_compiler: Compiler name and version.

  • python_cflags: Compiler flags used to compile Python.

  • python_executable: path to the Python executable

  • python_hash_seed: value of the PYTHONHASHSEED environment variable (random string or an int)

  • python_implementation: Python implementation. Examples: cpython, pypy, etc.

  • python_version: Python version, with the architecture (32 or 64 bits) if available, ex: 2.7.11 (64bit)

Memory metadata:

  • command_max_rss (int): Maximum resident set size in bytes (int) measured by Runner.bench_command().

  • mem_max_rss (int): Maximum resident set size in bytes (int). On Linux, kernel 2.6.32 or newer is required.

  • mem_peak_pagefile_usage (int): Get PeakPagefileUsage of GetProcessMemoryInfo() (of the current process): the peak value of the Commit Charge during the lifetime of this process. Only available on Windows.

CPU metadata:

  • cpu_affinity: if set, the process is pinned to the specified list of CPUs

  • cpu_config: Configuration of CPUs (ex: scaling governor)

  • cpu_count: number of logical CPUs (int)

  • cpu_freq: Frequency of CPUs

  • cpu_machine: CPU machine

  • cpu_model_name: CPU model name

  • cpu_temp: Temperature of CPUs

System metadata:

  • aslr: Address Space Layout Randomization (ASLR)

  • boot_time (str): Date and time of the system boot

  • hostname: Host name

  • platform: short string describing the platform

  • load_avg_1min (int or float >= 0): Load average figures giving the number of jobs in the run queue (state R) or waiting for disk I/O (state D) averaged over 1 minute

  • runnable_threads: number of currently runnable kernel scheduling entities (processes, threads). The value comes from the 4th field of /proc/loadavg: 1 in 0.20 0.22 0.24 1/596 10123 for example (596 is the total number of threads).

  • uptime (int or float >= 0): Duration since the system boot (float, number of seconds since boot_time)

Other:

  • perf_version: Version of the pyperf module

  • unit: Unit of values: byte, integer or second

  • calibrate_loops (int >= 1): number of loops computed in a loops calibration run

  • recalibrate_loops (int >= 1): number of loops computed in a loops recalibration run

  • calibrate_warmups (bool): True for runs used to calibrate the number of warmups

  • recalibrate_warmups (bool): True for runs used to recalibrate the number of warmups

pyperf JSON format

pyperf stores benchmark results as JSON in files. By default, the JSON is formatted to produce small files. Use the python3 -m pyperf convert --indent (...) command (see pyperf convert) to get readable (indented) JSON.

pyperf supports JSON files compressed by gzip: use gzip if filename ends with .gz.

Example of JSON, ... is used in the example for readability:

{
    "benchmarks": [
        {
            "runs": [
                {
                    "metadata": {
                        "date": "2016-10-21 03:14:19.670631",
                        "duration": 0.33765527700597886,
                    },
                    "warmups": [
                        [
                            1,
                            0.023075559991411865
                        ],
                        [
                            2,
                            0.022522017497976776
                        ],
                        [
                            4,
                            0.02247579424874857
                        ],
                        [
                            8,
                            0.02237467262420978
                        ]
                    ]
                },
                {
                    "metadata": {
                        "date": "2016-10-21 03:14:20.496710",
                        "duration": 0.7234010050015058,
                    },
                    "values": [
                        0.022752201875846367,
                        0.022529058374857414,
                        0.022569017250134493
                    ],
                    "warmups": [
                        [
                            8,
                            0.02249833550013136
                        ]
                    ]
                },
                ...
                {
                    "metadata": {
                        "date": "2016-10-21 03:14:52.549713",
                        "duration": 0.719920061994344,
                        ...
                    },
                    "values": [
                        0.022562820375242154,
                        0.022442164625317673,
                        0.02241712374961935
                    ],
                    "warmups": [
                        [
                            8,
                            0.02249412499986647
                        ]
                    ]
                }
            ]
        }
    ],
    "metadata": {
        "cpu_count": 4,
        "cpu_model_name": "Intel(R) Core(TM) i7-3520M CPU @ 2.90GHz",
        "description": "Telco decimal benchmark",
        "hostname": "selma",
        "loops": 8,
        "name": "telco",
        "perf_version": "0.8.2",
        "tags": ["numeric"],
        ...
    },
    "version": "1.0"
}

See also the jq tool: “lightweight and flexible command-line JSON processor”.