skip to content

A pull request claims its C++ JSON parser change is 30% faster on one Google Benchmark run; how do you check the speedup is real?

level: seniorimportance: should knowfreq 45%

answer

  1. one run is one sample
  2. same machine, same flags
  3. frequency scaling and noisy neighbours
  4. repetitions and interleaving
  5. a significance test on both builds

basics

~20 s

Build both versions identically, run them on a quiet machine with repetitions and warm-up, compare them with Google Benchmark's compare.py U test, check the inputs are realistic and nothing was optimized away, then explain the gain with a profiler.

solid answer

~40 s

One run is one sample from a noisy process: frequency scaling, turbo boost, other processes, core migration and even code layout can move a microbenchmark by tens of percent. I would build the base and the pull request with identical compiler and flags, run them on the same idle machine with the frequency governor fixed and the process pinned to a core, and use `--benchmark_repetitions` (20 or more) with `--benchmark_min_warmup_time` and `--benchmark_enable_random_interleaving`. Then `compare.py benchmarks` reports relative mean and median changes plus a Mann-Whitney U test p-value. I would also check the inputs are realistic documents rather than one small string, confirm the new code is not faster because its result became unused, and use a profiler or counters to explain where the 30% came from.

code

bash · 9 lines
bash
# Linux, idle machine: fix the frequency governor (from the library's reducing-variance notes)
sudo cpupower frequency-set --governor performance

# Base and pull-request binaries built with identical flags; compare.py ships in Google Benchmark's tools/
taskset -c 2 ./compare.py benchmarks ./parser_bench_base ./parser_bench_pr \
    --benchmark_filter=BM_Parse \
    --benchmark_repetitions=30 \
    --benchmark_min_warmup_time=0.5 \
    --benchmark_enable_random_interleaving

go deeper

for a junior

Recall that a single benchmark run is noisy and that both versions must be built and run the same way before comparing.

for a middle

Explain repetitions, warm-up and random interleaving in Google Benchmark, and what the mean, median and coefficient of variation in the aggregates tell you.

for a senior

Run the full procedure: quiet machine, compare.py with a U test, realistic inputs, a dead-code check, and a profiler-backed mechanism for the gain.

for a principal

Set the bar a performance claim must meet in review, and the reporting format that lets anyone reproduce it on the team's reference hardware.

## One run is one sample A Google Benchmark run reports a mean per iteration for one process on one machine at one moment. With `--benchmark_repetitions` left at its default of 1, there is no spread at all in the output, so nothing distinguishes a real 30% from a lucky run. Sources of variance that routinely move microbenchmarks by that much: - **CPU frequency scaling** and turbo features, which change the clock between and during runs; Google Benchmark prints `***WARNING*** CPU scaling is enabled, the benchmark real time measurements may be noisy and will incur extra overhead.` when it detects scaling. - **Other load**: a build, a browser or another job on the same host competing for cores and caches. - **Core migration** and heterogeneous cores running at different speeds. - **Address-space layout and code alignment**, which shift branch and cache behaviour between builds without any source change; the library also warns when ASLR is enabled. - **A debug build of the library itself**, flagged as `***WARNING*** Library was built as DEBUG. Timings may be affected.` ## A procedure that survives review 1. **Build both sides identically**: same compiler, same flags, same library versions, base commit and pull-request commit side by side. 2. **Quiet the machine**: the user guide's reducing-variance notes suggest the `performance` governor, disabling boost and pinning with `taskset`; dedicated hardware is better than a shared laptop. 3. **Repeat and warm up**: `--benchmark_repetitions=30` gives a distribution; `--benchmark_min_warmup_time` discards results while caches fill; `--benchmark_enable_random_interleaving` interleaves repetitions of different benchmarks so slow drift does not hit one of them only. 4. **Compare statistically**: `compare.py benchmarks <baseline> <contender> [benchmark options]` runs both binaries and prints relative changes in mean and median plus a p-value from a Mann-Whitney U test; a small p-value says the two distributions differ, the relative median says by how much. 5. **Check the effect size against the noise**: the coefficient of variation in the aggregates tells you whether a 30% change is far outside run-to-run spread or within it. ## Reading compare.py's output For each benchmark, `compare.py` prints rows whose `Time` and `CPU` columns are relative changes, computed as `(new - old) / |old|`, so `-0.30` means 30% faster: - **`_pvalue`**: the U test's p-value and the number of repetitions on each side; a value near zero means the two distributions very probably differ. - **`_mean`** and **`_median`**: the relative change in each; trust the median when a few repetitions were disturbed. - **`_stddev`** and **`_cv`**: how the spread changed; a new version that is faster on average but far more variable deserves a second look. - **`OVERALL_GEOMEAN`**: a geometric-mean summary across the compared benchmarks, useful as a one-line overview of a suite. ## Check what was measured, not only how | Question | Why it matters | |---|---| | Are the inputs realistic? | A parser tuned on one 200-byte string may be slower on the large, deeply nested or escape-heavy documents production sees; use `->Args` or a corpus of real payloads. | | Is the work still there? | A refactor that stops using the parse result can let dead-code elimination delete it; the speedup is then the optimizer's. | | Is the comparison fair? | Both versions must parse the same documents and produce the same output; a test that compares the results guards against a fast but wrong parser. | | Does throughput agree? | `state.SetBytesProcessed(...)` turns time into bytes per second, which is easier to sanity-check than nanoseconds. | ## Explain the gain A claimed speedup should come with a mechanism: fewer instructions, fewer cache misses, fewer allocations, fewer mispredicted branches. A sampling profiler on both builds shows which functions lost time; hardware counters show whether instructions or cycles per instruction changed. Google Benchmark can report counters such as `CYCLES` and `INSTRUCTIONS` itself through `--benchmark_perf_counters` when the library is built with libpfm support (the `BENCHMARK_ENABLE_LIBPFM` CMake option) on hardware with a performance monitoring unit. If no mechanism appears, treat the number as noise until a repeat proves otherwise. ## When the answer is "smaller than claimed" Most 30% claims shrink under this procedure — often to a real but smaller gain on some inputs and none on others. That is a useful result: the pull request description should then state the measured change, the inputs, the machine and the p-value, so the next reviewer can reproduce it.

  • The U test gives p = 0.001 but the median changed by only 2%. Should the pull request be merged as a speedup?
    The test says the difference is probably real, not that it matters. A 2% change on a microbenchmark may vanish in the whole program, and layout or alignment effects can produce consistent differences of that size. Report it honestly as a small, statistically significant improvement and judge it by the end-to-end metric the parser serves.
  • Why can moving unrelated code change a C++ microbenchmark's result?
    Code layout changes where loops and functions land relative to cache lines and branch-predictor structures, so a hot loop can become faster or slower without any change to its instructions. That is why one comparison of two builds proves little, and why repeated runs, several inputs and a mechanism from a profiler are needed before attributing a gain to the source change.

  • One benchmark run per version is enough if the difference is large
  • Comparing means alone shows whether a change is real
  • Running the base on a laptop and the change on a CI host is a fair comparison
  • A faster benchmark proves the parser is faster on production documents
  • Disabling frequency scaling removes all noise, so repetitions are unnecessary