skip to content

Why can a Google Benchmark of a C++ JSON parser report almost zero time per iteration, and how does benchmark::DoNotOptimize fix it?

level: middleimportance: must knowfreq 52%

answer

  1. the as-if rule
  2. an unused result is not observable
  3. the timed loop body
  4. force the result to exist
  5. barrier on the result, not the input

basics

~20 s

Under the as-if rule the optimizer may delete a call whose result is never used, so the timed loop measures nothing. benchmark::DoNotOptimize(result) forces the result into a register or memory, so the parse must actually run each iteration.

solid answer

~50 s

C++ compilers only have to reproduce a program's **observable behaviour** — volatile accesses, file output and the like. Timing is not part of it, so in `for (auto _ : state) parse(input);` with the result discarded, an optimizing build may remove the call entirely and Google Benchmark reports a fraction of a nanosecond. `benchmark::DoNotOptimize(total)`, given a local holding the result, is an empty inline-assembly statement that claims to read and write that value, so the compiler must produce it every iteration. It does not protect the computation itself: if the input is a compile-time constant and the function is pure, the compiler may compute the result once and time only the barrier. Keep the input opaque too, pass a non-const lvalue (v1.10.0 deprecates the `const&` overload), and use `benchmark::ClobberMemory()` when the work is a write to memory.

code

cpp · 30 lines
cpp
#include <benchmark/benchmark.h>
#include <string>

static long sum_json_array(const std::string& json) {  // stand-in for the parser under review
    long total = 0, cur = 0;
    for (char c : json) {
        if (c >= '0' && c <= '9') cur = cur * 10 + (c - '0');
        else { total += cur; cur = 0; }
    }
    return total;
}

static const std::string kInput = [] {
    std::string s = "[";
    for (int i = 0; i < 1000; ++i) s += std::to_string(i) + ",";
    return s + "0]";
}();

static void BM_ParseDiscarded(benchmark::State& state) {
    for (auto _ : state) sum_json_array(kInput);       // result unused: the optimizer may delete the call
}
static void BM_ParseKept(benchmark::State& state) {
    for (auto _ : state) {
        long total = sum_json_array(kInput);
        benchmark::DoNotOptimize(total);                // the result must exist, so the parse must run
    }
}
BENCHMARK(BM_ParseDiscarded);
BENCHMARK(BM_ParseKept);
BENCHMARK_MAIN();

go deeper

for a junior

Recall that an optimizer may delete work whose result is unused, and that Google Benchmark's DoNotOptimize keeps a result alive.

for a middle

Explain the as-if rule, what the state loop times, why the reported mean hides the deletion, and what DoNotOptimize and ClobberMemory each force.

for a senior

Show you also defeat constant folding and hoisting with opaque inputs, check throughput for plausibility and read the generated code before trusting a number.

for a principal

Make benchmark hygiene reviewable: shared helpers for opaque inputs, throughput counters on every parser benchmark, and a rule that suspiciously fast results are investigated.

## The as-if rule decides what survives The C++ standard describes an abstract machine and requires a conforming implementation to emulate only its **observable behaviour**: accesses through volatile glvalues, data written to files, and interactive input and output. Its footnote on the **as-if rule** says an implementation need not evaluate part of an expression if it can deduce that the value is not used and that no side effects affecting observable behaviour are produced. How long the program takes is not observable behaviour. A benchmark loop that calls a parser and throws the result away therefore contains, as far as the optimizer is concerned, nothing worth keeping. ## What Google Benchmark times A Google Benchmark function receives a `benchmark::State&` and runs its work inside `for (auto _ : state)`. The library times that loop and picks the iteration count itself: it keeps running until the minimum time (0.5 seconds by default, `--benchmark_min_time`) is reached, then reports total time divided by iterations. Code before the loop is setup and is not timed. Two consequences matter here: - If the loop body is empty after optimization, the library happily runs it billions of times and divides almost nothing by a huge count. - The reported number is a **mean per iteration**, so nothing in the output says the work disappeared; it only looks implausibly fast. In the example, the same parse shows 0.000 ns when its result is discarded and about 5 µs when it is kept. Built at `-O0`, both versions take the same time — which is why a debug build never reveals the problem. ## How DoNotOptimize and ClobberMemory work In Google Benchmark v1.10.0 on GCC and Clang, `benchmark::DoNotOptimize(value)` is an empty `asm volatile` statement whose operands say it reads and writes `value` in a register or memory, with a memory clobber. The compiler cannot see inside the assembly, so it must materialize `value` at that point on every iteration and must assume memory may have changed. `benchmark::ClobberMemory()` is a pure compiler fence — in v1.10.0, `std::atomic_signal_fence(std::memory_order_acq_rel)` — that forces pending writes to be performed; memory owned by a local object must first be escaped with `DoNotOptimize` on a pointer to it. | Tool | Use it when | Protects | |---|---|---| | `DoNotOptimize(result)` | the work returns a value | the result's existence, every iteration | | `DoNotOptimize(ptr)` then `ClobberMemory()` | the work writes into a buffer | the writes reaching memory | | `DoNotOptimize(input)` before the call | the input is known at compile time | the input from constant folding | ## What DoNotOptimize does not do The user guide is explicit that `DoNotOptimize(<expr>)` does **not** prevent optimizations on `<expr>` itself: 1. **Constant inputs fold.** If `parse` is visible and its argument is a compile-time constant, the compiler may compute the answer at compile time and pass the constant to the barrier. 2. **Loop-invariant calls hoist.** A function the compiler knows has no side effects, called with the same argument every iteration, can be evaluated once outside the loop; only the barrier repeats. 3. **The deprecated overload.** v1.10.0 marks `DoNotOptimize(Tp const&)` deprecated with the message that it can permit undesired compiler optimizations; store an expression's result in a local and pass that non-const lvalue. The defence is to make the input opaque — read it from a file, generate it at run time as the example does, or pass it through `DoNotOptimize` first — and to sanity-check the reported time: a parser that "handles" a 4 KB document in under a nanosecond did not run. ## Reading a suspicious result - Compare against a rough estimate: bytes per iteration divided by a plausible throughput. Report throughput with `state.SetBytesProcessed(...)` so absurd numbers stand out. - Inspect the generated code (the material on reading compiler output covers how) and check the call or loop is still there. - Vary the input size with `->Range(...)`: a time that does not grow with the input means the work was removed. - Watch the iteration count: a benchmark that ran an enormous number of iterations at a fraction of a nanosecond each, as in the example, is the classic signature of a deleted loop body. None of these checks is expensive, and together they catch nearly every benchmark the optimizer emptied before it reaches a pull request description.

  • Why is benchmarking a C++ change at -O0 no answer to the dead-code problem?
    At `-O0` nothing is deleted, but nothing is optimized either, so the numbers describe code nobody ships: inlining, vectorization and register allocation are what the change will actually run with. The measurement must use the release build's flags and defeat dead-code elimination with `DoNotOptimize` and opaque inputs instead.
  • When do you need benchmark::ClobberMemory() in addition to DoNotOptimize?
    When the work's effect is a store to memory rather than a returned value, such as filling a buffer. The compiler could drop stores nobody reads. Escape the buffer first with `DoNotOptimize(ptr)` so the compiler assumes someone may read it, then call `ClobberMemory()` after the work so the pending writes must actually happen.

  • The benchmark is fine because it ran at -O0, where nothing gets optimized away
  • DoNotOptimize stops the compiler from optimizing the expression passed to it
  • Wrapping the call in a timer guarantees the call is executed
  • A volatile copy of the result is the only way to keep the call
  • DoNotOptimize flushes the CPU caches so every iteration starts cold