skip to content

In a Rust criterion benchmark, why does b.iter(|| { fnv1a(&buf); }) report about 225 picoseconds, and how does std::hint::black_box fix it?

level: middleimportance: must knowfreq 35%

answer

  1. too fast to be real
  2. a pure call whose result is unused
  3. the closure returns unit
  4. opaque inputs, observed outputs
  5. a hint, not a guarantee

basics

~20 s

The closure throws away fnv1a's result, so the optimiser deletes the pure call and criterion times an empty loop. Return the value, which Bencher::iter passes to black_box, and wrap inputs in std::hint::black_box so they stay opaque.

solid answer

~40 s

`fnv1a` has no side effects, and the semicolon makes the closure return `()`, so in an optimised build the call can be removed: 225 ps for 65,536 bytes would be hundreds of terabytes per second. `std::hint::black_box`, stable since Rust 1.66, is an identity function the compiler is encouraged to treat as using its argument in any valid way. Criterion's `Bencher::iter` already passes each return value through it, so `b.iter(|| fnv1a(&buf))` measured about 59 µs. Inputs need it too: `fnv1a(b"interview")` with a constant input also collapsed to about 224 ps, while `fnv1a(black_box(b"interview"))` took about 2.4 ns. It is best effort only: the docs say programs cannot rely on it for correctness and that it gives no cryptographic guarantees, and it blocks reasoning about its output, not about the expression you pass in.

code

rust · 25 lines
rust
use std::hint::black_box;

use checksums::fnv1a;
use criterion::{Criterion, criterion_group, criterion_main};

fn pitfalls(c: &mut Criterion) {
    let buf: Vec<u8> = (0..64 * 1024).map(|i| (i % 251) as u8).collect();

    // The closure returns (): the pure call's result is unused.
    c.bench_function("discarded", |b| {
        b.iter(|| {
            fnv1a(&buf);
        })
    });
    // The closure returns the u32, which Bencher::iter passes to black_box.
    c.bench_function("returned", |b| b.iter(|| fnv1a(&buf)));
    // The input is a constant the optimiser can see.
    c.bench_function("const-input", |b| b.iter(|| fnv1a(b"interview")));
    c.bench_function("const-input-boxed", |b| {
        b.iter(|| fnv1a(black_box(b"interview")))
    });
}

criterion_group!(benches, pitfalls);
criterion_main!(benches);

go deeper

for a junior

Recall that the optimiser deletes unused pure work, and that std::hint::black_box around inputs and a returned result keeps a benchmark honest.

for a middle

Explain the two purposes of black_box, why criterion's iter already wraps the return value, and why a constant input can be folded even when the result is used.

for a senior

Spot the impossible number from the units, read the closure for a trailing semicolon or a literal input, and say what black_box does not promise.

for a principal

Treat a benchmark without a sanity check on its units as untrusted, and make returning the result and wrapping inputs part of the team's review checklist.

## What the optimiser is allowed to delete A benchmark runs in an optimised build (Cargo's bench profile inherits the release settings). The optimiser may remove any computation whose result nobody observes, as long as the program behaves the same. A checksum function like `fnv1a` reads memory and returns a `u32`; it prints nothing and writes nothing outside itself. Write the closure as `|| { fnv1a(&buf); }` and the semicolon discards the `u32`, so the closure returns `()`. Once `fnv1a` is inlined, the loop body does nothing observable and can be deleted. Criterion then faithfully times an empty loop: **about 225 picoseconds** per iteration on the test machine. A sanity check exposes it: 65,536 bytes in 225 ps would be roughly 290 TB/s, far beyond any memory system. A **constant input** is the second trap. `fnv1a(b"interview")` hashes a nine-byte literal the compiler can see, so it can compute the hash at compile time. That benchmark also reported about 224 ps even though the result was returned. ## What `std::hint::black_box` does `std::hint::black_box` (also `core::hint::black_box`) became stable in **Rust 1.66**. The standard library describes it as an identity function that the compiler is encouraged to treat as using its argument in any possible valid way. Its documentation lists two purposes: 1. It stops the compiler from making optimisations based on the **value it returns**: `black_box(&buf)` is a reference the compiler must treat as unknown. 2. It forces the **value passed in** to be computed, even if the return value of `black_box` is unused. Criterion's `Bencher::iter` calls `black_box(routine())` on every iteration, which covers purpose 2 for any closure that returns its result. You cover purpose 1 by wrapping inputs: `b.iter(|| fnv1a(black_box(&buf)))`. ## What it does not promise The documentation is explicit, and interviewers like candidates who know it: - It is provided on a **best-effort** basis, and how much it blocks can vary by platform and code-generation backend. - Programs **cannot rely on it for correctness**; it must not control critical program behaviour. - It gives **no guarantee for cryptographic or security purposes**, so it does not make code constant-time. - It affects how its **output** is treated, not its input expression: `black_box(5 * 10)` is likely compiled as `black_box(50)`. Write `black_box(5) * black_box(10)` to keep the multiplication. - Wrapping a function's **zero-sized result**, such as `()`, may have no effect. - During constant evaluation it is a no-op. ## Where to put it in a criterion benchmark | Closure | What the optimiser sees | Measured (one laptop) | |---|---|---| | `\|\| { fnv1a(&buf); }` | unused pure call | about 225 ps | | `\|\| fnv1a(&buf)` | result goes to criterion's `black_box` | about 59 µs | | `\|\| fnv1a(b"interview")` | constant input, foldable | about 224 ps | | `\|\| fnv1a(black_box(b"interview"))` | opaque input | about 2.4 ns | The habit that follows: - **Return** the value the benchmark computes instead of ending the closure with a semicolon. - **Wrap inputs** in `black_box`, especially constants and anything the compiler could see through. - **Check the units**: a time far below one byte per cycle for a byte-at-a-time loop means the work is gone. - **Use the standard one.** `criterion::black_box` was deprecated in criterion 0.6 and now forwards to `std::hint::black_box` with a `deprecated` warning. ## Where criterion already applies it Criterion uses `std::hint::black_box` in several places of its own, which is why returning a value is usually enough on the output side: - `Bencher::iter` passes each **return value** of the closure through it. - `iter_batched` and `iter_batched_ref` pass each **setup result** through it before timing and the collected outputs after timing. - `bench_with_input` passes the **input reference** through it before calling your closure, a change made in criterion 0.4. What criterion cannot see is a value captured by the closure from the surrounding function, such as `buf`, or a literal written inside it. Those are yours to wrap. ## Why it is not a bug in the compiler Deleting unobserved work is exactly what makes release builds fast. A benchmark is the unusual program that wants the work done for its own sake, so it has to say so. `black_box` is that statement, and because it is only a hint, the number still deserves the sanity check.

  • Why does b.iter(|| fnv1a(&buf)) measure real work without an explicit black_box on the result?
    Criterion's `Bencher::iter` passes every return value through `std::hint::black_box`, so a closure that returns the checksum keeps the call alive; it measured about 59 µs, the same as the version with a wrapped input. Wrapping the input is still the habit, because a constant input lets the compiler fold the call even when the result is returned.
  • Can std::hint::black_box be used to make cryptographic code constant-time?
    No. Its documentation says it offers no guarantees for cryptographic or security purposes and that programs cannot rely on it for correctness. It is a best-effort hint whose effect varies by platform and backend, and the docs add that no mechanism in the Rust language provides the guarantees constant-time cryptography needs.
  • What should you do with criterion::black_box in an older benchmark?
    Replace it with `std::hint::black_box`. Criterion 0.6 deprecated its own function, which now just forwards to the standard one, so under criterion 0.7 every call raises a deprecation warning that points to `std::hint::black_box()`. The standard function has been stable since Rust 1.66.

Timing a courier who notices that nobody will ever sign for the parcel and so never leaves the depot: the stopwatch is accurate, the trip never happened. black_box is the signature the optimiser has to assume someone is waiting for.

  • black_box guarantees that no optimisation can cross the call on any platform.
  • black_box(5 * 10) keeps the multiplication from being computed at compile time.
  • A sub-nanosecond time for 64 KiB just means the checksum is very fast.
  • black_box needs nightly or a crate because it is part of the unstable test crate.
  • black_box can keep cryptographic comparisons constant-time.

Assumes Rust 1.99 and criterion 0.7; std::hint::black_box has been stable since Rust 1.66, and criterion 0.6 deprecated its own copy.