skip to content

As lead of a C++ codebase, how would you design a CI performance gate that blocks regressions like a slower JSON parser without flaky failures?

level: principalimportance: should knowfreq 25%

answer

  1. what to measure, and where
  2. dedicated quiet runners
  3. base and change in the same job
  4. significance plus a threshold
  5. calibrate with A/A runs

basics

~20 s

Gate a few hot-path benchmarks on dedicated quiet runners, compare base and change in the same job with repetitions, fail only when a slowdown is both significant and above an A/A-calibrated threshold, and track history for slow drift.

solid answer

~50 s

I would gate a small, curated set of Google Benchmark cases on hot paths — the JSON parser among them — rather than everything, because each case costs runner time and adds noise. They run on **dedicated, quiet runners**, never shared CI VMs, and the job builds the base and the change and compares them on the same machine in the same job with `compare.py`, 20-30 repetitions and interleaving. The rule is two-sided: fail only when the U test is significant **and** the median slowed by more than a threshold such as 5%, set from A/A runs that compare a build with itself to measure the false-alarm rate. Results go to a history store as JSON so slow creep across many small changes is visible. Overrides need a stated reason and an owner. Deterministic proxies — allocation counts, instruction counts — complement wall-clock time because they are far less noisy.

go deeper

for a junior

Know that performance can regress silently and that some teams run benchmarks in CI to catch it.

for a middle

Explain why CI benchmarks need repetitions, a same-machine comparison and stored JSON output to be meaningful.

for a senior

Design the job: dedicated runners, base-versus-change in one job, compare.py with a U test, warm-up, interleaving and deterministic proxy metrics.

for a principal

Own the trade-offs: sensitivity against false alarms, runner cost and pipeline time, threshold calibration, overrides and drift tracking, and whether a blocking gate is worth it at all.

## What a gate has to achieve A performance gate is worth having only if engineers trust it. It must catch a real regression — a parser change that costs 15% on large documents — and it must almost never fail a change that did nothing, because a gate that cries wolf gets disabled or overridden by habit. Every design choice below trades **sensitivity** against **false alarms** and **cost**. ## What to measure - **A curated set**, not every benchmark: hot paths whose speed users feel, such as JSON parsing, message encoding and the order-matching loop. Each case should have realistic inputs and pass dead-code checks. - **Throughput where it fits**: `state.SetBytesProcessed(...)` makes parser results comparable across input sizes. - **Deterministic proxies beside time**: allocation counts per operation and, in a Google Benchmark build with libpfm support, hardware counters through `--benchmark_perf_counters` such as `INSTRUCTIONS`. They move far less run to run, so a tight threshold is possible on them while wall-clock time gets a looser one. ## Where and how to run 1. **Dedicated hardware.** Shared CI VMs have noisy neighbours and variable clocks; reserve a few identical bare-metal or isolated runners, with the frequency governor fixed and boost disabled. 2. **Base and change in one job.** Build both commits with identical flags and compare them on the same machine within minutes: `compare.py benchmarks ./bench_base ./bench_pr --benchmark_repetitions=25`. Comparing against a number stored weeks ago imports every machine and environment change since. 3. **Repetitions and interleaving.** 20-30 repetitions per case with `--benchmark_enable_random_interleaving` and a warm-up; output saved with `--benchmark_out=<file>` and `--benchmark_out_format=json`. 4. **Time budget.** Decide how many minutes the gate may add to a pull request and choose cases to fit; run the long tail of benchmarks nightly instead. ## The decision rule | Rule | False alarms | Missed regressions | |---|---|---| | Fail if the mean is slower by any amount | very many | few | | Fail if slower by more than X% | some, on noisy cases | small regressions | | Fail if the U test is significant | many tiny but real changes | few | | Fail if significant **and** slower by more than X% | few | regressions below X% | The last row is the usual choice. X is not a guess: **calibrate it with A/A runs**, comparing a build with itself many times, and set X above the spread you observe for each case. Noisy cases get wider thresholds or move to the nightly job. ## Catching slow drift A per-change gate cannot see ten changes that each cost 1%. Store every run's JSON with the commit, plot each case over time, and alert when a moving window drifts past a budget. That turns "the parser got 12% slower this quarter" into a bisectable list of commits instead of a surprise. ## The JSON parser regression, through the gate Suppose a change speeds up small documents but makes the token buffer grow on large ones. The gate's parser case runs a corpus with both sizes; `compare.py` on the dedicated runner reports the large-document benchmark 18% slower with a p-value near zero, above the case's calibrated 6% threshold, so the check fails. The allocation-count proxy for the same case jumped from zero to three per parse, which points straight at the cause. The author either reserves from a size hint, or, if the trade is deliberate, applies the override label with the measured numbers and gets the benchmark owner's approval. ## Ownership and overrides - **Overrides are allowed but visible**: a label or annotation with the reason, such as a deliberate trade of speed for correctness, and a named approver. - **Each benchmark has an owner** who triages its failures and keeps its inputs realistic. - **Flaky cases are quarantined**, not deleted: moved to the nightly job until their noise is understood. ## What it costs, and when it is worth it Dedicated runners, longer pull-request pipelines and the time to maintain benchmarks and thresholds are real costs. They pay off where latency or throughput is part of the product — a trading engine, a game's frame budget, a parsing library others depend on. A codebase where performance rarely matters may do better with a nightly trend job and no blocking gate at all. Choosing between those, and revisiting the choice as the product changes, is the lead's call.

  • Why compare base and change in the same CI job instead of against a stored baseline number?
    A stored number was measured on some machine at some time; firmware updates, kernel changes, thermal state and other tenants have moved since. Building both commits and running them on the same runner minutes apart cancels most of that drift, so the comparison isolates the code change. Stored results remain useful for long-term trend charts.
  • How do A/A runs help set a performance gate's threshold?
    An A/A run compares a build with itself, so every difference it reports is noise. Repeating it many times shows each benchmark's natural spread and how often a given rule would fail an unchanged build. Set the threshold so that false-alarm rate is acceptably low, and move cases whose spread is too wide to a nightly job.

  • Run every benchmark on every pull request on the shared CI runners
  • Compare each change against a baseline number recorded months ago
  • Fail the build whenever the mean is slower at all
  • A statistically significant slowdown always deserves to block a merge
  • Flaky performance tests should simply be deleted