Benchmarking

Benchmarking#

ppbcc benchmark discovers Google Benchmark executables in a build folder, runs them, and consolidates their JSON reports into a single tidy CSV that P3 Analysis consumes.

Note

This workflow expects the build tree of the performance-portability-benchmark repository, or any other project whose targets emit Google Benchmark JSON.

Running benchmarks#

Always start with --dry-run to check which executables the regexes match:

cd <build>
ppbcc benchmark -p src -H "INTEL Data Center GPU Max 1550" \
  -r "vec_.*" "matMul_.*" "nbody_.*" "polyhedral_.*" -x ".*_cpp" --dry-run

Drop --dry-run to actually run them. Each executable writes a JSON report next to itself; existing reports are reused unless --force is given.

Consolidating existing reports#

With --skip-benchmark nothing is executed — --path/--regex then select the JSON reports directly. This is how the result CSVs in the benchmark repository’s results/ folder are produced, one per hardware platform:

ppbcc benchmark -p ./nvidia-rtx5080 -r ".*\.json" --skip-benchmark \
  -H "NVIDIA RTX5080" -o "Results_NVIDIA_RTX5080"
ppbcc benchmark -p ./nvidia-gh200 -r ".*\.json" --skip-benchmark \
  -H "NVIDIA GH200" -o "Results_NVIDIA_GH200"
ppbcc benchmark -p ./amd-mi210 -r ".*\.json" --skip-benchmark \
  -H "AMD Instinct MI210" -o "Results_AMD_Instinct_MI210"

The -H/--hardware value ends up in the Hardware column and is what ppbcc p2analysis/p3analysis treat as a platform when computing performance portability — so keep it consistent across runs.

Output format#

The consolidated CSV contains one row per benchmark observation:

Column

Meaning

Benchmark Problem

Problem name, e.g. MatrixMultiplication

Paradigm

Implementation label, e.g. Kokkos

Description

Variant detail, e.g. SharedMemory

Precision

Floating-point precision (32 or 64)

Hardware

Platform identifier from -H

Problem Size

Workload size

Iterations

Google Benchmark repetition count

Wall Clock Time, CPU Time, Kernel Time, …

Measured timings; Time Unit records their unit and the loader normalises them to a common base

A concrete row:

Benchmark Problem,Paradigm,Description,Precision,Hardware,Problem Size,Iterations,Wall Clock Time,...,Time Unit
MatrixMultiplication,OpenACC,,32,NVIDIA RTX5080,32.0,31702.0,21740.16,...,ns

Python API#

from pathlib import Path
from ppbcc.benchmark import find_files, load_reports, run_benchmarks

executables = find_files(
    [Path("build/src")], [r"nbody_.*"], require_executable=True, exclude=[r".*_cpp"]
)
reports = run_benchmarks(executables)
frame = load_reports(reports, hardware="NVIDIA RTX5080")

See API — ppbcc.benchmark for the full signatures.