Kernel Profiling & Roofline

ppbcc profile runs a profiler over a set of benchmark executables, one after another, consolidates the hardware counters into one tidy CSV, and draws a roofline model from them. --profiler selects the backend:

ncu is the default and needs no instrumentation: it attaches from the outside and replays every launch it sees. likwid reads the counters from inside the application, around the regions marked in the benchmark source, which keeps several kernels of one call together in a single row. All backends write the same columns, so the CSV and the plot are identical in shape, and --from-csv merges the tables of several of them into one roofline.

Note

Nsight Compute and LIKWID both read their counters through CUPTI, which needs a current CUDA context. Executables whose paradigm builds its own (*_ocl, *_boost, *_vulkan, *_slang_vulkan) or that never reach the GPU (*_cpp) produce no report; they are reported as such and skipped. ppbcc.profiling.nsys is the backend for those.

Building the executables for profiling

A profiler replays every kernel launch it sees, so a benchmark that sweeps a range of problem sizes and repeats each of them takes hours under ncu and produces thousands of near-identical kernels. The performance-portability-benchmark repository therefore has a dedicated build switch:

cmake --preset cuda-llvm-profiling      # = cuda-llvm plus -DPPB_PROFILING=ON
cmake --build build-cuda-llvm-profiling -j

PPB_PROFILING=ON reduces every executable to a single input running a single iteration and repetition, so profiling one binary takes about a second and every implementation is profiled on the same input:

Benchmark problem	Input under PPB_PROFILING
Vector addition	\(10^8\) elements
Matrix multiplication	\(4096 \times 4096\)
N-body simulation	\(10^5\) particles
Polyhedral gravity model	SHAPE_SFM_3M_v20180804 (3.1 M faces)

It is the largest input except where that would defeat the profiler: one Nsight Compute replay pass at \(16384^2\) has to snapshot several GB around every launch, and Eros’ polyhedral kernels are only a few microseconds long — too short for a counter- or sample-based tool to resolve.

PPB_PROFILING=ON also switches on PPB_ENABLE_NVTX, which names the kernels (see Regions below), and compiles the targets with line tables so a profiler can map an instruction back to a source line. The optimization flags are untouched: a roofline measured from a differently optimized binary describes a different program.

Profiling

As with Benchmarking, start with --dry-run:

ppbcc profile -b build-cuda-llvm-profiling -p src -r "polyhedral_.*" --dry-run

Drop it to profile. Each run writes <report-dir>/<executable>.ncu-rep next to the Google-Benchmark report <report-dir>/<executable>.json — the profiler does not know which paradigm a kernel was written in, so the JSON report supplies the paradigm and the precision:

ppbcc profile -b build-cuda-llvm-profiling -p src -r "polyhedral_.*" \
  -d profiling -H "NVIDIA RTX5080" -o Profiling_NVIDIA_RTX5080 --roofline

Existing reports are reused unless --force is given, and --skip-profile runs nothing at all and only re-parses the reports already in --report-dir. That splits nicely into a collection pass per toolchain (--no-csv, writing into a shared --report-dir) and one consolidation pass over all of them.

Note

Google Benchmark re-executes its process to switch off address space randomization, and Nsight Compute hangs on some binaries (matMul_kokkos, matMul_omp) when it follows that exec. The ncu backend therefore starts every run through setarch <arch> -R, so the re-exec never happens. setarch ships with util-linux; where it is missing, the run keeps the re-exec and a warning is logged. -O aslr=on opts out.

What is measured

The collected metric set follows Nvidia’s own roofline recipe:

• Work is counted from executed SASS instructions, per precision: \(\mathrm{FLOP} = \mathrm{add} + \mathrm{mul} + 2 \cdot \mathrm{fma}\). All three precisions are always reported (FLOP FP32, FLOP FP64, FLOP FP16); --precision selects which one the roofline uses, with auto following the precision the binary was built with.

• Traffic is read at a selectable level of the hierarchy — --memory-level picks dram (default), l2 or l1.

• Both ceilings are measured, not looked up. Every .peak_sustained metric is a per-cycle rate; multiplied with the clock the corresponding unit actually ran at during the kernel it becomes an absolute rate. The highest value observed across all profiled kernels becomes the roof.

The LIKWID backend

LIKWID reads the counters through its NvMarker API, from regions that the benchmark source opens itself. Build the benchmark with the markers compiled in:

cmake --preset cuda-llvm-profiling -B build-likwid \
  -DPPB_ENABLE_LIKWID=ON -DLIKWID_ROOT="$LIKWID_PREFIX"
cmake --build build-likwid -j

Then point ppbcc profile at it. LIKWID exposes no peak_sustained counters, so the two ceilings have to be supplied; they are a property of the hardware, and an ncu run on the same machine measures them:

ppbcc profile --profiler likwid -b build-likwid -p src -r "polyhedral_.*" \
  -O lib="$LIKWID_PREFIX/lib" -d ../profiling-likwid \
  --peak-performance 5.74e13 --peak-bandwidth 9.59e11 \
  -H "NVIDIA RTX5080" --roofline

Three properties of LIKWID 5.5.1 are worth knowing.

The likwid-perfctr wrapper is bypassed. It programs the counters from a separate process, which fails on Blackwell with CUPTI_ERROR_INVALID_PARAMETER regardless of the CUDA version. ppbcc therefore sets LIKWID_NVMON_GPUS, LIKWID_NVMON_EVENTS and LIKWID_NVMON_FILEPATH and runs the binary directly, so the instrumented process programs its own counters.

Every executable runs twice. The SMSP counters (floating-point instructions) and the DRAM counter cannot be programmed in the same pass, so the events are split into the two groups of EVENT_GROUPS and merged afterwards.

The two runs do not cost the same. Programming three SMSP counters perturbs a kernel far more than a single DRAM counter – on the 16384² matrix multiplication the same region takes about 36s in the first group and 3.6s in the second. The counters themselves are unaffected by that overhead, but the region time is, so the reported duration is the smallest of the two, which is the least perturbed measurement of the kernel. Run at -v to see the spread.

The counters are corrected by a factor of two. On this hardware LIKWID reports every Nvidia counter at exactly half its true value. The factor was established against kernels with an analytically known instruction count: for FADD, FMUL, FFMA and DRAM_BYTES_SUM alike, ncu reproduces the analytic value exactly and LIKWID returns half of it, independently of the counter domain and of how many launches a region contains. COUNTER_SCALE compensates for it. Note that arithmetic intensity is a ratio of two equally scaled counters and is therefore unaffected either way; only the absolute rates need the correction.

Warning

NvMarker needs a current CUDA context, so it sees the same paradigms Nsight Compute does. OpenCL and Vulkan build their own contexts and produce no marker file. The markers around those kernels are still compiled in, so the gap shows up as an explicit warning rather than silently.

The Nsight Systems backend

--gpu-metrics-devices programs the GPU’s performance monitors in time-based sampling mode: device-wide, no context filter, no kernel boundaries. That is what makes it the only backend here that sees OpenCL and Vulkan. What it gives up is attribution — a sample knows when, not which kernel — and the benchmark buys that back: --trace=nvtx records the marked regions with the same clock as the samples, so a window is named by the region it falls in and the stretches where compute warps are in flight inside it are the kernel.

ppbcc profile --profiler nsys -b build-cuda-llvm-profiling -p src \
  -r "matMul_ocl$" "matMul_vulkan$" -d ../profiling-nsys \
  -O iterations=20 --peak-performance 5.74e13 --peak-bandwidth 9.592e11 \
  --analytic-flop "matMul_.*=137438953472" -H "NVIDIA RTX5080"

Three options carry the method.

--analytic-flop

The metric sets available on consumer hardware carry pipe utilisations, not instruction counts, so the FLOP count has to come from the work model. It is not a guess where it matters: Nsight Compute counts exactly \(2 \cdot M \cdot N \cdot K\) for every CUDA-backed matrix multiplication.

-O iterations

A lazily initialising backend compiles pipelines and fills buffers on its first call, and the sampler cannot tell that from a kernel launch. Every binary is therefore profiled twice, at 1 and at N iterations; the difference over N-1 iterations cancels the setup both runs paid.

-O activity-ratio

Some runtimes move buffers with a compute shader rather than a copy engine, which shows up as compute-active windows that are not the kernel — and, because the marked region encloses everything the benchmark times, inside the same region. A window below this fraction of the busiest one’s occupancy in that region is dropped.

--peak-bandwidth is required: the sampler reports DRAM traffic as a percentage of peak, so the peak is what turns it into bytes. Both ceilings are properties of the hardware, and an ncu run on the same machine measures them.

The Nsight Graphics backend

GPU Trace reads the same performance monitors as Nsight Compute — as counter sums rather than samples — and it is the one Nvidia tool that does so for a Vulkan workload, which makes it the independent check on the sampled numbers. Its unit of attribution is the trace: with no swapchain there are no frames, and the per-regime table stays empty. The region is bounded by submit index instead, and -O submit=auto traces the first -O probes submissions and keeps the one with the most compute cycles. A Vulkan submission carries no NVTX range, so this is the one backend whose rows are unnamed.

Warning

auto picks the busiest submission, which is the wrong one whenever the first call also does the lazy initialisation — it is then longer than the steady-state dispatch. Pass -O submit=<index> explicitly in that case.

Regions

A profiler names a kernel whatever the compiler called it, which for several paradigms is nothing useful: AdaptiveCpp launches four kernels all called __acpp_sscp_kernel, Kokkos eleven called Kokkos::cuda_parallel_launch_local_memory. The benchmark therefore names the work itself. src/common/Marker.h brackets each phase with an NVTX range — matmul for the matrix multiplication, init and evaluate for the polyhedral gravity — and every backend reports which range a row belongs to in the Region column. -DPPB_PROFILING=ON turns that on.

Everything outside those ranges is the runtime setting itself up: Kokkos’ architecture query, desul’s lock arrays, a framework’s buffer staging. Those launches say nothing about the algorithm, so they are dropped before the CSV is written; --all-kernels keeps them. An executable with no range at all is kept unchanged, with a warning, since that means the binary was built without PPB_ENABLE_NVTX rather than that its kernels are uninteresting.

--region restricts the plot to some of them; the CSV always keeps every region:

# The polyhedral kernel without its one-off initialization.
ppbcc profile ... --roofline --region evaluate

Aggregation

One region may launch several kernels. --aggregate decides what a point is:

sum (default)

One point per implementation and region: work, traffic and kernel time of every launch in it are added up.

dominant

One point per implementation and region, its longest-running kernel only.

none

One point per kernel launch.

Output

The CSV holds one row per profiled kernel launch: the identifying columns (problem, paradigm, precision, hardware, executable, region, kernel, grid/block size), the derived roofline quantities (duration, FLOP, traffic, arithmetic intensity, performance, and both ceilings), and behind them every raw ncu metric. Every quantity carries its unit in the header, e.g. Arithmetic Intensity [FLOP/Byte] or dram__bytes.sum.peak_sustained [Byte/cycle]; --from-csv reads files with and without units.

--skip-profile does not need the ncu CLI: without one on the PATH, the reports are read through Nsight Compute’s ncu_report Python module, which the macOS host application ships as well.

The roofline chart uses the same paradigm colors as P3 Analysis, so a point can be matched to its paradigm across both figures.