API — ppbcc.profiling

Module Overview

Batch kernel profiling and roofline analysis.

Four backends produce the same tidy table, differing only in what they can see: ppbcc.profiling.runner drives Nsight Compute from the outside, ppbcc.profiling.nsys samples the GPU’s performance monitors device-wide (the only one that sees OpenCL and Vulkan), ppbcc.profiling.ngfx traces a Vulkan queue submission, and ppbcc.profiling.likwid reads the counters inside regions marked in the benchmark source.

Metric Set

Nsight Compute metric set and the roofline quantities derived from it.

The metric names follow Nvidia’s own roofline recipe: floating-point work is counted from the executed SASS instructions (an FMA counts twice), memory traffic is read at a selectable level of the hierarchy, and the two ceilings are measured rather than looked up in a datasheet — every .peak_sustained metric is a per-cycle rate that becomes an absolute rate once multiplied with the clock frequency that unit actually ran at during the kernel.

ppbcc.profiling.metrics.TIMING_METRICS = ['gpu__time_duration.sum', 'sm__cycles_elapsed.avg', 'sm__cycles_elapsed.avg.per_second']

Wall-clock duration and the clock rates the ceilings are scaled with.

ppbcc.profiling.metrics.FLOP_METRICS = ['sm__sass_thread_inst_executed_op_fadd_pred_on.sum', 'sm__sass_thread_inst_executed_op_fmul_pred_on.sum', 'sm__sass_thread_inst_executed_op_ffma_pred_on.sum', 'sm__sass_thread_inst_executed_op_dadd_pred_on.sum', 'sm__sass_thread_inst_executed_op_dmul_pred_on.sum', 'sm__sass_thread_inst_executed_op_dfma_pred_on.sum', 'sm__sass_thread_inst_executed_op_hadd_pred_on.sum', 'sm__sass_thread_inst_executed_op_hmul_pred_on.sum', 'sm__sass_thread_inst_executed_op_hfma_pred_on.sum']

Executed floating-point SASS instructions, per precision and operation.

ppbcc.profiling.metrics.TENSOR_METRICS = ['sm__inst_executed_pipe_tensor.sum']

Tensor-core instructions; reported for context, never counted as FLOPs.

ppbcc.profiling.metrics.MEMORY_METRICS = ['dram__bytes.sum', 'dram__bytes.sum.peak_sustained', 'dram__cycles_elapsed.avg.per_second', 'lts__t_bytes.sum', 'lts__t_bytes.sum.peak_sustained', 'lts__cycles_elapsed.avg.per_second', 'l1tex__t_bytes.sum', 'l1tex__t_bytes.sum.peak_sustained', 'l1tex__cycles_elapsed.avg.per_second']

Bytes moved and the achievable bytes/cycle, per level of the hierarchy.

ppbcc.profiling.metrics.PEAK_METRICS = ['sm__sass_thread_inst_executed_op_ffma_pred_on.sum.peak_sustained', 'sm__sass_thread_inst_executed_op_dfma_pred_on.sum.peak_sustained', 'sm__sass_thread_inst_executed_op_hfma_pred_on.sum.peak_sustained']

Achievable floating-point instructions per cycle (the compute ceilings).

ppbcc.profiling.metrics.LAUNCH_METRICS = ['launch__grid_size', 'launch__block_size', 'launch__waves_per_multiprocessor', 'sm__throughput.avg.pct_of_peak_sustained_elapsed', 'gpu__dram_throughput.avg.pct_of_peak_sustained_elapsed']

Launch configuration, useful to explain a point that sits far below the roof.

ppbcc.profiling.metrics.ROOFLINE_METRICS = ['gpu__time_duration.sum', 'sm__cycles_elapsed.avg', 'sm__cycles_elapsed.avg.per_second', 'sm__sass_thread_inst_executed_op_fadd_pred_on.sum', 'sm__sass_thread_inst_executed_op_fmul_pred_on.sum', 'sm__sass_thread_inst_executed_op_ffma_pred_on.sum', 'sm__sass_thread_inst_executed_op_dadd_pred_on.sum', 'sm__sass_thread_inst_executed_op_dmul_pred_on.sum', 'sm__sass_thread_inst_executed_op_dfma_pred_on.sum', 'sm__sass_thread_inst_executed_op_hadd_pred_on.sum', 'sm__sass_thread_inst_executed_op_hmul_pred_on.sum', 'sm__sass_thread_inst_executed_op_hfma_pred_on.sum', 'sm__inst_executed_pipe_tensor.sum', 'dram__bytes.sum', 'dram__bytes.sum.peak_sustained', 'dram__cycles_elapsed.avg.per_second', 'lts__t_bytes.sum', 'lts__t_bytes.sum.peak_sustained', 'lts__cycles_elapsed.avg.per_second', 'l1tex__t_bytes.sum', 'l1tex__t_bytes.sum.peak_sustained', 'l1tex__cycles_elapsed.avg.per_second', 'sm__sass_thread_inst_executed_op_ffma_pred_on.sum.peak_sustained', 'sm__sass_thread_inst_executed_op_dfma_pred_on.sum.peak_sustained', 'sm__sass_thread_inst_executed_op_hfma_pred_on.sum.peak_sustained', 'launch__grid_size', 'launch__block_size', 'launch__waves_per_multiprocessor', 'sm__throughput.avg.pct_of_peak_sustained_elapsed', 'gpu__dram_throughput.avg.pct_of_peak_sustained_elapsed']

Every metric collected by ppbcc profile.

ppbcc.profiling.metrics.PRECISIONS = {'fp16': ('h', 'FP16'), 'fp32': ('f', 'FP32'), 'fp64': ('d', 'FP64')}

Precision key -> (ncu instruction infix, human-readable label).

ppbcc.profiling.metrics.MEMORY_LEVELS = {'dram': ('dram__bytes', 'DRAM'), 'l1': ('l1tex__t_bytes', 'L1/TEX'), 'l2': ('lts__t_bytes', 'L2')}

Memory level key -> (ncu counter base, human-readable label).

ppbcc.profiling.metrics.MEMORY_LEVEL_CLOCK = {'dram': 'dram__cycles_elapsed.avg.per_second', 'l1': 'l1tex__cycles_elapsed.avg.per_second', 'l2': 'lts__cycles_elapsed.avg.per_second'}

dram__bytes is clocked by dram__cycles_elapsed, the others by the unit that shares their name prefix.

ppbcc.profiling.metrics.DERIVED_UNITS = {'Arithmetic Intensity': 'FLOP/Byte', 'Block Size': 'threads/block', 'Duration': 's', 'FLOP': 'FLOP', 'FLOP FP16': 'FLOP', 'FLOP FP32': 'FLOP', 'FLOP FP64': 'FLOP', 'Grid Size': 'blocks', 'Memory Traffic': 'Byte', 'Peak Bandwidth': 'Byte/s', 'Peak Performance': 'FLOP/s', 'Performance': 'FLOP/s'}

Units of the derived roofline columns, as written into the CSV header.

ppbcc.profiling.metrics.column_unit(column)

Return the unit of a profiling column.

Raw metrics follow Nsight Compute’s naming scheme, from which their unit follows (checked against IMetric.unit() of the reports): .sum of bytes or instructions is a count, .peak_sustained the same per clock cycle, .per_second a clock rate, pct_of_* a percentage.

Parameters:

column (str) – Column name without a unit.

Returns:

The unit, or None for identifying and textual columns.

Return type:

str | None

ppbcc.profiling.metrics.flop_columns(precision)

Return the add/mul/fma metric names of one precision.

Parameters:

precision (str) – Key of PRECISIONS (fp32, fp64 or fp16).

Returns:

The add, mul and fma metric names, in that order.

Return type:

tuple[str, str, str]

ppbcc.profiling.metrics.peak_flop_column(precision)

Return the peak_sustained FMA metric name of one precision.

Parameters:

precision (str) – Key of PRECISIONS.

Returns:

The metric name holding the achievable FMA instructions per cycle.

Return type:

str

Profiler Execution

Batch execution of Nvidia Nsight Compute (ncu) over benchmark binaries.

ppbcc.profiling.runner.REPORT_SUFFIX = '.ncu-rep'

File name suffix Nsight Compute appends to --export.

ppbcc.profiling.runner.aslr_prefix()

Return the launcher that runs a child with ASLR disabled.

Google Benchmark’s main calls MaybeReenterWithoutASLR, which execv``s the process to get reproducible timings. Nsight Compute attaches to the pre-exec process; with ``--target-processes all it then also attaches to the re-executed one, and on some binaries (matMul_kokkos, matMul_omp) the profiler afterwards spins at 100 % CPU with the application suspended and never finishes. Starting the process with ASLR already off makes Google Benchmark skip the re-exec, and the same runs complete in seconds.

Returns:

The setarch ... -R argument vector, or an empty list if setarch is unavailable (the run then simply keeps the re-exec).

Return type:

list[str]

ppbcc.profiling.runner.find_ncu(explicit=None)

Locate the ncu executable.

Parameters:

explicit (str | None) – User-supplied path, or None to search PATH.

Returns:

Path to the profiler executable.

Raises:

FileNotFoundError – If no ncu executable can be found.

Return type:

str

ppbcc.profiling.runner.profile_command(ncu, target, report, metrics=None, benchmark_report=None, extra_ncu_args=None, extra_target_args=None, disable_aslr=True)

Build the ncu command line for one benchmark executable.

Parameters:

• ncu (str) – Profiler executable.

• target (Path) – Benchmark binary to profile.

• report (Path) – Destination report; .ncu-rep is appended by the profiler.

• metrics (list[str] | None) – Metrics to collect, defaulting to ROOFLINE_METRICS.

• benchmark_report (Path | None) – Where the target should write its Google-Benchmark JSON report. It carries the paradigm and precision of the run, which the profiler output itself does not know about.

• extra_ncu_args (list[str] | None) – Additional profiler arguments.

• extra_target_args (list[str] | None) – Additional arguments for the benchmark binary.

• disable_aslr (bool) – Whether to start the profiler through aslr_prefix(), which stops Google Benchmark from re-executing the process under the profiler.

Returns:

The argument vector to execute.

Return type:

list[str]

ppbcc.profiling.runner.run_profiles(executables, output_dir, ncu, force=False, metrics=None, extra_ncu_args=None, extra_target_args=None, stream_output=False, timeout=None, disable_aslr=True)

Profile every executable, one after another.

Each run produces <output_dir>/<executable name>.ncu-rep next to the Google-Benchmark report <output_dir>/<executable name>.json, so the profiling artefacts carry the same names as the binaries they came from.

Parameters:

• executables (list[Path]) – Benchmark binaries to profile.

• output_dir (Path) – Directory the reports are written to; created if missing.

• ncu (str) – Profiler executable.

• force (bool) – Re-profile even when a report already exists.

• metrics (list[str] | None) – Metrics to collect, defaulting to ROOFLINE_METRICS.

• extra_ncu_args (list[str] | None) – Additional profiler arguments.

• extra_target_args (list[str] | None) – Additional arguments for the benchmark binaries.

• stream_output (bool) – Whether to forward the profiler output to the logger.

• timeout (float | None) – Wall-clock limit per executable in seconds, or None for no limit. A run that hits the limit is skipped, and the batch goes on.

• disable_aslr (bool) – Whether to launch through aslr_prefix().

Returns:

The reports that exist after the batch, in execution order.

Return type:

list[Path]

Report Parsing

Read .ncu-rep reports and turn them into tidy roofline tables.

ppbcc.profiling.reports.SM_CLOCK = 'sm__cycles_elapsed.avg.per_second'

Clock rate the compute ceiling is scaled with.

ppbcc.profiling.reports.DURATION_METRIC = 'gpu__time_duration.sum'

Kernel duration reported by ncu, in nanoseconds.

ppbcc.profiling.reports.load_reports(reports, ncu='ncu', hardware='', precision='auto', memory_level='dram')

Load profiler reports into one tidy table, one row per kernel launch.

Parameters:

• reports (list[Path]) – Paths to .ncu-rep files.

• ncu (str | None) – Profiler executable used to re-import the reports, or None to read them through Nsight Compute’s ncu_report Python module instead (a host without the CLI, such as macOS).

• hardware (str) – Free-form hardware identifier stored in the Hardware column (e.g. "NVIDIA RTX5080").

• precision (str) – auto to follow the precision the binary was built with, or an explicit key of PRECISIONS.

• memory_level (str) – Level of the memory hierarchy the arithmetic intensity is measured against; a key of MEMORY_LEVELS.

Returns:

One row per profiled kernel launch, with the raw metrics and the derived roofline quantities. Empty if nothing could be loaded.

Return type:

DataFrame

ppbcc.profiling.reports.with_units(frame)

Append the unit to every column that has one, for writing the CSV.

Parameters:

frame (DataFrame) – A profiling table with bare column names.

Returns:

A copy whose headers read <name> [<unit>].

Return type:

DataFrame

ppbcc.profiling.reports.without_units(frame)

Strip the units with_units() appended, for reading a CSV back.

Parameters:

frame (DataFrame) – A profiling table read from a CSV, with or without units.

Returns:

The table with bare column names.

Return type:

DataFrame

ppbcc.profiling.reports.load_csv(paths)

Concatenate consolidated profiling CSVs into one table.

No single profiler covers every paradigm in this benchmark: Nsight Compute reads counters only inside a CUDA context, so Vulkan and OpenCL have to be measured with other tools. Each tool writes the same columns, and this is how their tables become one roofline.

Parameters:

paths (list[Path]) – Paths to CSVs written by ppbcc profile.

Returns:

The concatenated table, deduplicated on (executable, kernel id). Empty if nothing could be read.

Return type:

DataFrame

ppbcc.profiling.reports.filter_regions(frame, keep_unlabelled=False)

Keep only the launches that happened inside a named region.

The benchmark brackets the work it is about – matmul, init, evaluate – with the NVTX ranges of src/common/Marker.h. Everything outside them is the runtime setting itself up: Kokkos’ architecture query, desul’s lock-array initialization, a framework’s buffer staging. Those launches say nothing about the algorithm and would distort any aggregate over an executable, so they do not belong in the table.

An executable that carries no region at all is kept as it is, with a warning: it was built without PPB_ENABLE_NVTX, profiled without --nvtx, or measured by a backend that cannot see NVTX ranges – Nsight Graphics traces a queue submission, which carries none. Dropping every one of its rows would hide that.

Parameters:

• frame (DataFrame) – Table returned by one of the backends’ load_reports.

• keep_unlabelled (bool) – Keep the launches outside every region as well.

Returns:

The table without the unlabelled launches.

Return type:

DataFrame

ppbcc.profiling.reports.select_regions(frame, patterns)

Keep only the regions whose name matches one of the given patterns.

Parameters:

• frame (DataFrame) – Table returned by filter_regions().

• patterns (list[str]) – Regular expressions matched (re.search) against the region name; an empty list keeps every region.

Returns:

The table restricted to the matching regions.

Return type:

DataFrame

ppbcc.profiling.reports.aggregate_kernels(frame, mode='sum')

Collapse the kernel launches of each executable into plot points.

A region – matmul, init, evaluate – is the unit of work the benchmark named, so it is also the unit the points are grouped by. An implementation with a separate initialization kernel therefore contributes two points, one per phase, rather than one point mixing them.

Parameters:

• frame (DataFrame) – Table returned by load_reports().

• mode (str) – sum adds up the work and the time of every kernel launch of a region, which places one point per implementation and region; dominant keeps only the longest-running kernel of each; none keeps every launch.

Returns:

The rows to plot, with intensity and performance recomputed for sum.

Return type:

DataFrame

LIKWID Backend

LIKWID NvMarker backend: GPU counters read from regions marked in the source.

Where Nsight Compute attaches to a process from the outside and replays every kernel launch it sees, LIKWID reads the counters from inside the application: src/common/Marker.h in the benchmark repository opens a region around the kernel each implementation already times, and CUPTI reads the counters when that region closes. The two tools therefore answer slightly different questions – ncu reports one row per kernel launch, LIKWID one row per marked region – and the region is the more useful unit whenever an implementation reaches the GPU through a runtime that launches several kernels per call.

Two properties of LIKWID 5.5.1 shape this module.

The wrapper is bypassed. likwid-perfctr programs the counters from a separate daemon process, which fails on this hardware (cuptiProfilerGetCounterAvailability -> CUPTI_ERROR_INVALID_PARAMETER, independent of the CUDA version). The NvMarker API is therefore driven directly: the environment variables below make the instrumented binary program its own counters, and no wrapper is involved.

The counters are read in two passes. The SMSP domain (floating-point instructions) and the DRAM domain cannot be programmed at the same time – createConfigImage rejects the combination – so every executable is run once per EVENT_GROUPS entry and the results are merged afterwards.

ppbcc.profiling.likwid.MARKER_SUFFIX = '.likwid-marker'

File name suffix of a parsed marker file.

ppbcc.profiling.likwid.EVENT_GROUPS: dict[str, list[str]] = {'flops': ['SMSP_SASS_THREAD_INST_EXECUTED_OP_FADD_PRED_ON_SUM', 'SMSP_SASS_THREAD_INST_EXECUTED_OP_FMUL_PRED_ON_SUM', 'SMSP_SASS_THREAD_INST_EXECUTED_OP_FFMA_PRED_ON_SUM'], 'memory': ['DRAM_BYTES_SUM']}

Event groups, collected in separate runs. The keys name the run, the values are the LIKWID event names in the order their counts appear in the marker file.

ppbcc.profiling.likwid.COUNTER_SCALE = 2.0

LIKWID 5.5.1 reports every Nvidia counter at exactly half its hardware value on this machine. The factor was established against kernels with an analytically known instruction count and cross-checked with Nsight Compute: for FADD, FMUL, FFMA and DRAM_BYTES_SUM alike, ncu reproduces the analytic value exactly and LIKWID reports half of it. The factor is independent of the counter domain and of the number of launches inside a region, so it is a constant scale rather than an averaging error, and correcting for it here restores agreement with ncu. Note that arithmetic intensity is a ratio of two equally scaled counters and is therefore unaffected either way.

ppbcc.profiling.likwid.event_string(group, precision)

Build the LIKWID_NVMON_EVENTS value for one event group.

Parameters:

• group (str) – Key of EVENT_GROUPS.

• precision (str) – Key of PRECISIONS.

Returns:

Comma-separated EVENT:GPUn pairs, in counter order.

Return type:

str

ppbcc.profiling.likwid.parse_marker_file(path)

Parse a LIKWID NvMarker output file.

The format is three blocks of whitespace-separated fields, as written by nvmon_markerClose in LIKWID’s src/libnvctr.c:

<gpus> <regions> <groups>
<region index>:<tag>-<group id>            (one line per region)
<region index> <group id> <gpu id> <calls> <time> <events> <value>...

Note that the header counts GPUs first and regions second, and that the region index leads the data lines while the group id follows it. Both are easy to mistake for one another while every file holds a single region and a single group, where the two are indistinguishable.

Parameters:

path (Path) – Marker file written by the instrumented binary.

Returns:

Region tag -> {"time": seconds, "calls": n, "values": [...]}.

Return type:

dict[str, dict[str, float]]

ppbcc.profiling.likwid.run_profiles(executables, output_dir, likwid_lib=None, gpu=0, precision='fp32', force=False, extra_target_args=None, stream_output=False, timeout=None)

Run every executable once per event group and collect its marker files.

Parameters:

• executables (list[Path]) – Instrumented benchmark binaries.

• output_dir (Path) – Directory the marker files are written to.

• likwid_lib (Path | None) – lib directory of the LIKWID installation, prepended to LD_LIBRARY_PATH when given.

• gpu (int) – Index of the GPU to read counters from.

• precision (str) – Key of PRECISIONS.

• force (bool) – Re-profile even when marker files already exist.

• extra_target_args (list[str] | None) – Additional arguments for the benchmark binaries.

• stream_output (bool) – Forward the binary’s output to the logger.

• timeout (float | None) – Wall-clock limit per run in seconds, or None for no limit.

Returns:

The executables that produced at least one marker file, in execution order. The files themselves are named <executable>.<group>.likwid-marker and stay separate per group – each carries its own header block, so they cannot be concatenated.

Return type:

list[Path]

ppbcc.profiling.likwid.find_profiled(report_dir)

List the executables a previous run left marker files for.

Used by --skip-profile, which re-reads an existing batch instead of running one. Only the file name matters downstream, so the returned paths are name stubs, not runnable binaries.

Parameters:

report_dir (Path) – Directory holding the marker files.

Returns:

One path per executable, sorted by name.

Return type:

list[Path]

ppbcc.profiling.likwid.load_reports(executables, report_dir, hardware='', precision='auto', peak_performance=None, peak_bandwidth=None, scale=2.0)

Turn the marker files of a batch into the tidy roofline table.

The table has the same columns as the Nsight Compute one, so both backends feed the same CSV writer and the same plot. The one structural difference is the row granularity: a row is a marked region, not a kernel launch, and its kernel column carries the region tag.

Parameters:

• executables (list[Path]) – Benchmark binaries whose marker files should be read.

• report_dir (Path) – Directory holding the marker files and the Google-Benchmark reports written alongside them.

• hardware (str) – Label for the hardware column.

• precision (str) – The precision whose counters were collected – pass the same value that run_profiles() ran with. auto falls back to the precision the binary was built with, which is only correct when the two happen to agree.

• peak_performance (float | None) – Compute ceiling in FLOP/s. LIKWID exposes no peak_sustained counters, so the ceiling has to be supplied; it is a property of the hardware, not of the measuring tool.

• peak_bandwidth (float | None) – Memory ceiling in bytes/s, supplied for the same reason.

• scale (float) – Correction applied to every counter, see COUNTER_SCALE.

Returns:

One row per marked region, ordered as PROFILE_COLUMN_LIST.

Return type:

DataFrame

Nsight Systems Backend

Nsight Systems backend: roofline quantities for paradigms without a CUDA context.

Nsight Compute and LIKWID both read their counters through CUPTI, which needs a current CUDA context. OpenCL and Vulkan build their own, so neither tool sees them – which leaves four of this benchmark’s paradigms without a roofline point.

Nsight Systems closes that gap from the other side. --gpu-metrics-devices programs the GPU’s performance monitors in time-based sampling mode: the counters are read device-wide at a fixed frequency, with no context filter and no kernel boundaries, so the samples cover whatever the GPU is doing regardless of which API submitted it. What the sampler gives up is attribution – a sample knows when, not which kernel – and this module buys it back from the benchmark itself. --trace=nvtx records the NVTX ranges of src/common/Marker.h with the same clock as the samples, so every sample falls inside a named region (matmul, init, evaluate) or outside all of them. Within a region, the stretches where compute warps are in flight are the kernel.

Two quantities come out of such a window:

• its length, which is the kernel duration, and

• the integral of the DRAM read and write throughputs over it, which is the memory traffic.

The third roofline quantity, the FLOP count, is the one the sampler cannot supply: the metric sets available on this hardware carry pipe utilisations, not instruction counts. It comes from the analytic work model instead (--analytic-flop), which for these benchmarks is not a guess – Nsight Compute reproduces it exactly on every CUDA-backed implementation of the same problem.

ppbcc.profiling.nsys.REPORT_SUFFIX = '.nsys-rep'

Suffix of the profiler’s own report.

ppbcc.profiling.nsys.DATABASE_SUFFIX = '.sqlite'

Suffix of the SQLite export the report is read from.

ppbcc.profiling.nsys.COMPUTE_ACTIVITY_METRIC = 'Compute Warps in Flight [Throughput %]'

Metric names the window detection and the traffic integral need. Nsight Systems numbers its metrics per metric set, so they are resolved by name from TARGET_INFO_GPU_METRICS rather than hard-coded.

ppbcc.profiling.nsys.ACTIVITY_THRESHOLD = 1.0

Compute activity above this percentage counts as “a kernel is running”.

ppbcc.profiling.nsys.WINDOW_GAP_SECONDS = 0.0005

Samples separated by less than this are treated as one window. The sampler reports a per-interval average, so a kernel whose occupancy dips between two waves produces a short run of empty samples inside a single launch. 0.5 ms was calibrated against binaries whose kernel time is known independently: it recovers 41.43 ms for matMul_ocl (the application’s own clGetEventProfilingInfo delta is 41.45 ms), 38.09 ms for matMul_vulkan (38.91 ms) and 44.17 ms for matMul_cuda (Nsight Compute: 43.11 ms), while anything below 0.2 ms splits every one of them.

ppbcc.profiling.nsys.MINIMUM_WINDOW_SECONDS = 0.0001

Shortest window that can be a kernel rather than a bootstrap launch.

ppbcc.profiling.nsys.ACTIVITY_RATIO = 0.7

A window whose mean compute occupancy is below this fraction of the busiest window’s is treated as data movement rather than as the kernel. Some runtimes move their buffers with a compute shader instead of a copy engine – Kompute brackets every matrix-multiplication dispatch with two such phases – and those show up as compute-active windows that the roofline must not count. The separation is wide: the dispatch runs at ~96 % occupancy, the transfers at ~49 %.

ppbcc.profiling.nsys.REFERENCE_INFIX = '.ref'

Infix of the reference run of the two-point measurement, see run_profiles().

ppbcc.profiling.nsys.find_nsys(explicit=None)

Locate the nsys executable.

Parameters:

explicit (str | None) – User-supplied path, or None to search PATH.

Returns:

Path to the profiler executable.

Raises:

FileNotFoundError – If no nsys executable can be found.

Return type:

str

ppbcc.profiling.nsys.profile_command(nsys, target, report, metric_set, frequency, benchmark_report=None, extra_target_args=None)

Build the nsys profile command line for one benchmark executable.

Parameters:

• nsys (str) – Profiler executable.

• target (Path) – Benchmark binary to profile.

• report (Path) – Destination report; the profiler appends .nsys-rep.

• metric_set (str) – GPU metric set alias, e.g. gb20x for a Blackwell consumer part. nsys profile --gpu-metrics-set=help lists them.

• frequency (int) – Sampling frequency in Hz.

• benchmark_report (Path | None) – Where the target writes its Google-Benchmark JSON report, which carries the paradigm and precision.

• extra_target_args (list[str] | None) – Additional arguments for the benchmark binary.

Returns:

The argument vector to execute.

Return type:

list[str]

ppbcc.profiling.nsys.run_profiles(executables, output_dir, nsys, metric_set, frequency=100000, iterations=1, force=False, extra_target_args=None, stream_output=False, timeout=None)

Profile every executable, one after another, and export each report.

With iterations above one every executable is profiled twice, once with a single iteration and once with iterations. The sampler cannot tell a kernel launch from the pipeline compilation and buffer uploads that a lazily initialising backend does on its first call – both are compute work on the same GPU, and they end up inside the same window. Two runs separate them: the constant setup cost appears in both, so subtracting the single-iteration measurement from the many-iteration one and dividing by the difference in iteration count leaves the per-iteration kernel alone.

Parameters:

• executables (list[Path]) – Benchmark binaries to profile.

• output_dir (Path) – Directory the reports and SQLite exports are written to.

• nsys (str) – Profiler executable.

• metric_set (str) – GPU metric set alias.

• frequency (int) – Sampling frequency in Hz.

• iterations (int) – Iterations of the many-iteration run; 1 disables the two-point measurement.

• force (bool) – Re-profile even when a database already exists.

• extra_target_args (list[str] | None) – Additional arguments for the benchmark binaries.

• stream_output (bool) – Forward the profiler output to the logger at TRACE level.

• timeout (float | None) – Wall-clock limit per run in seconds, or None.

Returns:

The SQLite databases of the main run, in execution order. The reference run of each is named <executable>.ref.sqlite next to it.

Return type:

list[Path]

ppbcc.profiling.nsys.find_profiled(report_dir)

List the SQLite exports a previous batch left behind.

Parameters:

report_dir (Path) – Directory holding the exports.

Returns:

The databases, sorted by name.

Return type:

list[Path]

ppbcc.profiling.nsys.analytic_flop(name, rules)

Resolve the analytic FLOP count of one row.

Parameters:

• name (str) – What the rules are matched against. The callers pass <executable>[<region>], e.g. polyhedral_ocl[evaluate], so a rule can address one phase of a binary – the polyhedral init kernel computes normals and segment vectors, not the gravity model, and the work model of evaluate does not describe it. A rule naming only the executable still matches every one of its regions.

• rules (list[str]) – <regex>=<value> rules, in order; the first match wins.

Returns:

The FLOP count, or None if no rule matches.

Return type:

float | None

ppbcc.profiling.nsys.load_reports(databases, report_dir, hardware='', precision='auto', peak_performance=None, peak_bandwidth=None, flop_rules=None, iterations=1, activity_ratio=0.7)

Turn the sampled reports of a batch into the tidy roofline table.

The columns are the ones the Nsight Compute backend writes, so both feed the same CSV writer and the same plot. A row is one named region of one implementation – matmul, or init and evaluate.

Parameters:

• databases (list[Path]) – SQLite exports of the main run.

• report_dir (Path) – Directory holding the Google-Benchmark reports written alongside them; they supply paradigm, precision and problem.

• hardware (str) – Label for the hardware column.

• precision (str) – auto to follow the precision the binary was built with, or an explicit key of PRECISIONS.

• peak_performance (float | None) – Compute ceiling in FLOP/s. Nsight Systems reports percentages of peak but not the peak itself, so it has to be supplied; it is a property of the hardware, and an ncu run on the same machine measures it.

• peak_bandwidth (float | None) – Memory ceiling in bytes/s, supplied for the same reason and needed to turn the sampled percentages into bytes.

• flop_rules (list[str] | None) – <regex>=<value> rules supplying the analytic FLOP count per executable, see analytic_flop().

• iterations (int) – Iteration count the main run used. Above one, the .ref report next to each database is subtracted per region and the remainder divided by how many more times that region was entered, which cancels the one-time setup both runs paid; see run_profiles().

• activity_ratio (float) – Occupancy threshold that separates the kernel from data-movement phases, see ACTIVITY_RATIO.

Returns:

One row per (executable, NVTX region), ordered as PROFILE_COLUMN_LIST.

Return type:

DataFrame

Nsight Graphics Backend

Nsight Graphics GPU Trace backend: hardware counters for the Vulkan paradigms.

Nsight Systems (ppbcc.profiling.nsys) sees Vulkan work, but only through device-wide sampling: it reports what fraction of peak each unit was running at, averaged over a sampling interval. Nsight Graphics’ GPU Trace Profiler reads the same performance monitors the way Nsight Compute does – as counter sums over a bounded region – and it is the one Nvidia tool that does so for a Vulkan workload. That makes it the independent check on the sampled numbers, which is the role it plays here.

Its unit of attribution is the trace, not the dispatch: with no swapchain there are no frames, and the per-regime table it exports stays empty. The region is bounded by submit index instead: --start-after-submits k --limit-to-submits n traces exactly the k-th submission. A PPB_PROFILING binary issues a handful of submits – upload, dispatch, download – so --ngfx-submit auto simply traces each of the first few and keeps the one that spent the most cycles in compute.

As with Nsight Systems, the FLOP count is not measured: the Blackwell metric set carries pipe utilisations rather than instruction counts, so it comes from the analytic work model (--analytic-flop).

ppbcc.profiling.ngfx.FRAME_TIME_FILE = PosixPath('BASE/FRAME.xls')

Where the tool writes the exported tables, relative to --output-dir.

ppbcc.profiling.ngfx.FRAME_METRICS_FILE = PosixPath('BASE/GPUTRACE_FRAME.xls')

Trace-wide counter sums, one name<TAB>value line each.

ppbcc.profiling.ngfx.DRAM_METRIC = 'dram__sectors.sum'

Total DRAM traffic of the trace. Despite the name the exported value is in bytes: dividing it by the frame time reproduces the accompanying dram__sectors.sum.per_second in GB/s and the matching pct_of_peak_sustained_elapsed against a 960 GB/s peak.

ppbcc.profiling.ngfx.COMPUTE_CYCLES_METRIC = 'gr__compute_cycles_active_queue_sync.sum'

Cycles the graphics/compute engine spent on compute work, used by --ngfx-submit auto to recognise the dispatch among the submissions.

ppbcc.profiling.ngfx.MAXIMUM_DURATION_MS = 9000

Nsight Graphics refuses longer traces.

ppbcc.profiling.ngfx.find_ngfx(explicit=None)

Locate the ngfx executable.

Parameters:

explicit (str | None) – User-supplied path, or None to search PATH and the default installation directory.

Returns:

Path to the Nsight Graphics CLI.

Raises:

FileNotFoundError – If it cannot be found.

Return type:

str

ppbcc.profiling.ngfx.profile_command(ngfx, target, working_directory, output_dir, architecture, submit, submits=1, metric_set=0, benchmark_report=None)

Build the ngfx command line for one benchmark executable.

Parameters:

• ngfx (str) – Nsight Graphics CLI.

• target (Path) – Benchmark binary to profile.

• working_directory (Path) – Directory the binary is launched in; the benchmarks resolve their input meshes relative to it.

• output_dir (Path) – Directory the trace and the exported tables land in.

• architecture (str) – Architecture whose metric set is selected, e.g. "Blackwell GB20x". ngfx --help-all lists the names.

• submit (int) – Index of the first traced queue submission.

• submits (int) – How many submissions to trace.

• metric_set (int) – Index of the metric set for architecture.

• benchmark_report (Path | None) – Where the target writes its Google-Benchmark JSON report, which carries the paradigm and precision.

Returns:

The argument vector to execute.

Return type:

list[str]

ppbcc.profiling.ngfx.run_profiles(executables, output_dir, ngfx, working_directory, architecture, submit='auto', probes=4, metric_set=0, force=False, stream_output=False, timeout=None)

Trace every executable and keep the submission that did the compute work.

Parameters:

• executables (list[Path]) – Benchmark binaries to profile.

• output_dir (Path) – Directory the per-executable trace directories are created in.

• ngfx (str) – Nsight Graphics CLI.

• working_directory (Path) – Directory the binaries are launched in.

• architecture (str) – Architecture whose metric set is selected.

• submit (str) – auto to probe the first probes submissions and keep the one with the most compute cycles, or an explicit index.

• probes (int) – How many submissions auto tries.

• metric_set (int) – Index of the metric set for architecture.

• force (bool) – Re-trace even when a result already exists.

• stream_output (bool) – Forward the tool’s output to the logger at TRACE level.

• timeout (float | None) – Wall-clock limit per trace in seconds, or None.

Returns:

The per-executable directories holding the kept trace, in execution order.

Return type:

list[Path]

ppbcc.profiling.ngfx.find_profiled(report_dir)

List the traces a previous batch left behind.

Probe directories carry a .submitN suffix and are skipped; only the kept trace of each executable is returned.

Parameters:

report_dir (Path) – Directory holding the per-executable trace directories.

Returns:

The directories, sorted by name.

Return type:

list[Path]

ppbcc.profiling.ngfx.load_reports(traces, report_dir, hardware='', precision='auto', peak_performance=None, peak_bandwidth=None, flop_rules=None)

Turn the traces of a batch into the tidy roofline table.

Parameters:

• traces (list[Path]) – Per-executable trace directories.

• report_dir (Path) – Directory holding the Google-Benchmark reports; they supply paradigm, precision and problem.

• hardware (str) – Label for the hardware column.

• precision (str) – auto or an explicit key of PRECISIONS.

• peak_performance (float | None) – Compute ceiling in FLOP/s; Nsight Graphics reports percentages of peak but not the peak itself.

• peak_bandwidth (float | None) – Memory ceiling in bytes/s, for the same reason.

• flop_rules (list[str] | None) – <regex>=<value> rules supplying the analytic FLOP count.

Returns:

One row per traced submission, ordered as PROFILE_COLUMN_LIST.

Return type:

DataFrame

Command Line Interface

Command-line interface for batch kernel profiling.

Four backends are available, selected with --profiler. They differ in what they can see, not in what they report: each one writes the same columns, so their tables concatenate into one roofline (--from-csv).

• ncu – Nsight Compute. Counts instructions and bytes per kernel launch, but only inside a CUDA context.

• nsys – Nsight Systems. Samples the GPU’s performance monitors device-wide, which is what makes OpenCL and Vulkan visible.

• ngfx – Nsight Graphics GPU Trace. Sums the same counters over a Vulkan queue submission; an independent cross-check for the Vulkan paradigms.

• likwid – reads counters inside the regions marked in the benchmark source.

Backend-specific settings are passed as -O <name>=<value> rather than as one flag each; --help lists the names each backend accepts.

ppbcc.profiling.cli.BACKEND_OPTIONS: dict[str, tuple[str, object, object, str]] = {'activity-ratio': ('nsys', <class 'float'>, 0.7, "compute-active windows whose mean occupancy is below this fraction of the busiest window's in the same region are treated as data movement rather than as the kernel. Some runtimes move buffers with a compute shader instead of a copy engine, and those phases sit inside the same marked region as the kernel"), 'architecture': ('ngfx', <class 'str'>, 'Blackwell GB20x', "architecture whose metric set is selected; 'ngfx --help-all' lists the names"), 'aslr': ('ncu', <function <lambda>>, False, "'on' keeps address space randomization. Off by default, which stops Google Benchmark from re-executing the process under the profiler; Nsight Compute otherwise hangs on some binaries (matMul_kokkos, matMul_omp)"), 'frequency': ('nsys', <class 'int'>, 100000, 'sampling frequency in Hz'), 'gpu': ('likwid', <class 'int'>, 0, 'index of the GPU the counters are read from'), 'iterations': ('nsys', <class 'int'>, 20, 'iterations of the many-iteration run. The sampler cannot tell a kernel launch from the pipeline compilation and buffer uploads a lazily initialising backend does on its first call, so every executable is profiled twice, once with a single iteration and once with N; subtracting the first from the second cancels the setup both paid. 1 disables the second run and leaves the setup cost in the result'), 'lib': ('likwid', <class 'pathlib.Path'>, None, 'lib directory of the LIKWID installation, prepended to LD_LIBRARY_PATH'), 'metric-set': ('nsys', <class 'str'>, 'gb20x', "GPU metric set alias; must match the architecture ('gb20x' for a Blackwell consumer part). 'nsys profile --gpu-metrics-set=help' lists the aliases"), 'metrics': ('ncu', <function <lambda>>, None, 'comma-separated metrics to collect instead of the roofline set'), 'ncu-arg': ('ncu', 'append', [], 'extra argument passed through to ncu; repeatable'), 'probes': ('ngfx', <class 'int'>, 4, "how many submissions 'submit=auto' tries"), 'submit': ('ngfx', <class 'str'>, 'auto', "queue submission to trace. Without a swapchain there are no frames, so the traced region is bounded by submit index instead. 'auto' traces the first 'probes' submissions and keeps the one that spent the most cycles in compute")}

name -> (backend, converter, default, help). Keeping them out of the flag namespace is what stops the parser from growing a flag per backend per knob; a name that does not belong to the selected backend is an error, so a typo cannot pass silently.

Type:

Backend settings reachable through -O

ppbcc.profiling.cli.parse_options(pairs, profiler)

Turn the -O name=value arguments into the selected backend’s settings.

Parameters:

• pairs (list[str]) – The raw name=value strings, in order.

• profiler (str) – The backend the settings have to belong to.

Returns:

Every option of that backend, defaults filled in.

Raises:

ValueError – On a malformed pair, an unknown name, a name belonging to a different backend, or a value the converter rejects.

Return type:

dict[str, object]

ppbcc.profiling.cli.build_parser()

Build the profile command parser.

Returns:

Configured argument parser.

Return type:

ArgumentParser

ppbcc.profiling.cli.main(argv=None)

Run the profile command.

Parameters:

argv (list[str] | None) – Command-line arguments, or None to read sys.argv.

Returns:

Process exit status.

Return type:

int