A C++ JSON parser change cuts the Google Benchmark mean by 30%, yet production p99 parse time got worse; how do you measure and explain the tail?
answer
- a mean per iteration hides the tail
- repetition statistics are not call percentiles
- time each call into a preallocated array
- realistic input mix
- profile the slow inputs on their own
basics
~20 sGoogle Benchmark reports a mean per iteration, which hides rare slow calls. Time each parse individually over a realistic input mix, compute p99 and max, find which inputs are slow, then profile those inputs alone to see why.
solid answer
~40 sA Google Benchmark result is total time divided by iterations, and `ComputeStatistics` with repetitions only summarizes repetition means, so neither shows the 99th percentile of individual calls. To see the tail I time every call: read a monotonic clock around each parse, store the durations in a vector reserved before the loop, and compute p50, p99 and max with `std::nth_element`. The input must mirror production — mostly small documents with the occasional huge or escape-heavy one — because the tail usually comes from rare inputs or rare events: a buffer that grows and allocates, a page fault, a slow path for Unicode escapes. A sampling profiler on the mixed workload mostly shows the common path; instead I capture the slow inputs, profile them in isolation, and check for allocations on the slow path.
code
cpp · 30 lines#include <algorithm>
#include <chrono>
#include <cstdio>
#include <string>
#include <vector>
static unsigned long parse(const std::string& json) { // stand-in for the parser; unsigned wraps, no UB
unsigned long total = 0;
for (char c : json) total = total * 10 + static_cast<unsigned long>(c == ',' ? 0 : c - '0');
return total;
}
int main() {
std::string small(400, '7'), big(100000, '7'); // 1 call in 50 parses a large document
std::vector<double> ns;
ns.reserve(100000); // no allocation inside the timed loop
volatile unsigned long sink = 0;
for (int i = 0; i < 100000; ++i) {
const std::string& doc = (i % 50 == 49) ? big : small;
auto t0 = std::chrono::steady_clock::now();
sink = sink + parse(doc);
auto t1 = std::chrono::steady_clock::now();
ns.push_back(std::chrono::duration<double, std::nano>(t1 - t0).count());
}
double mean = 0;
for (double x : ns) mean += x / ns.size();
auto pct = [&](double p) { auto it = ns.begin() + static_cast<long>(p * (ns.size() - 1));
std::nth_element(ns.begin(), it, ns.end()); return *it; };
std::printf("mean %.0f ns, p50 %.0f ns, p99 %.0f ns, max %.0f ns\n", mean, pct(0.50), pct(0.99), pct(1.0));
}go deeper
Recall that an average hides slow outliers, and that latency is described with percentiles such as p50 and p99.
Explain why Google Benchmark's mean and repetition statistics do not give per-call percentiles, and how to record each call's time without disturbing it.
Link tail samples to inputs, allocations and page faults, profile slow inputs in isolation, and explain why aggregated sampling profiles miss rare paths.
Decide which percentiles a component's performance contract states and how releases are checked against them, not only against the mean.
## Why the benchmark could not see it Google Benchmark answers "how long does one iteration take on average?" It runs the loop body many times and reports total time divided by the iteration count. Two mechanisms that look like they might show a tail do not: - **Repetitions** (`--benchmark_repetitions`) rerun the whole measurement and report mean, median, standard deviation and coefficient of variation **of the repetition means**. - **Custom statistics** (`->ComputeStatistics("max", ...)`) apply your function to that same vector of repetition results, so a "p99" statistic over 30 repetitions is the 99th percentile of 30 averages, not of individual calls. A change that makes the common case 30% faster while making one call in fifty much slower can lower the mean and raise the p99 at the same time. The mean is a ratio of totals; the tail is a property of the distribution. ## Measuring per-call latency in C++ The direct method is to time each call and keep every sample: 1. Reserve a `std::vector<double>` for all samples before the loop, so recording never reallocates inside the measurement loop and disturbs the calls being timed. 2. Read a monotonic clock before and after each call (the material on clocks covers `std::chrono::steady_clock` and its resolution). 3. Keep the result alive — a `volatile` sink or `benchmark::DoNotOptimize` — so the call is not removed. 4. After the loop, compute percentiles with `std::nth_element`, and report p50, p99, p99.9 and max beside the mean. In the example, a mix where one call in fifty parses a large document gives a mean around 2.4 µs, a median around 0.4 µs and a p99 around 100 µs on this machine: three numbers that tell three different stories. When one call is too short for the clock's resolution, time small batches instead and say so. Reading the clock also costs time, typically tens of nanoseconds, which is negligible against a 100 µs parse but not against a 50 ns one. Report the max separately from p99.9: a single sample far above everything else is often interference from the machine, such as the thread being preempted, rather than the parser; one run of the example had a 14 ms max that no parse in it could explain. ## Where C++ parser tails come from | Source | Why it is rare | How to confirm | |---|---|---| | Large or unusual inputs | most documents are small | log input size and shape with each slow sample | | Heap allocation on a slow path | a buffer grows or a map rehashes only past a threshold | count calls to `operator new` per parse | | Page faults on fresh memory | first touch of newly allocated pages | compare first-run and warm-run latencies | | Branchy slow paths | escapes, Unicode, deep nesting | profile the slow inputs alone | | The machine | preemption, frequency changes, interrupts | a max far above p99.9 that does not correlate with input | ## What a sampling profiler shows, and what it hides A sampling profiler such as Linux `perf` (`perf record`, then `perf report`) interrupts the program at intervals and records the stack, so time is attributed in proportion to how often each function is on the CPU. That is ideal for the mean and poor for a tail: - A path taken by 1% of calls collects only its share of samples, so it barely shows in an aggregated profile even when each of those calls is very slow. - Time spent off the CPU — blocked on a lock, on I/O or waiting for the scheduler — is not sampled as on-CPU work. - Inlining folds small functions into their callers, and frames without frame pointers or unwind information produce truncated stacks; the build needs debug information. The practical approach is to split the problem: use the per-call recording to identify which inputs or conditions are slow, then profile a workload made only of those inputs, where the slow path is the common path. ## Reporting the result A useful answer to the reviewer reports p50, p99 and max for the old and new parser on the same input mix, names the mechanism behind the regression — for example, the new parser's token buffer now grows on large documents — and proposes a fix, such as reserving from a size hint, so the mean gain is kept without the tail.
- Why can't you average per-run p99 values from several runs into one p99?Percentiles do not compose by averaging: the mean of several p99 values is not the p99 of the combined samples. Keep the raw samples, or a mergeable histogram of them, and compute the percentile over everything; the material on performance percentiles covers the aggregation rules in depth.
- How would you check whether the new parser's tail comes from heap allocation?Count calls to the global `operator new` per parse in a test build and record the count with each latency sample. If the slow samples line up with non-zero counts, the tail is allocation: a buffer growing or a container rehashing. Reserving from a size hint, or reusing buffers across parses, usually removes it.
- If the mean improved, the p99 cannot have got worse
- ComputeStatistics over repetitions gives the per-call 99th percentile
- A sampling profiler of the mixed workload will show the slow path clearly
- Average the p99 of each run to get the overall p99
- The maximum latency is always caused by the parser's own code