skip to content

In Rust on the stable toolchain, how do you write a benchmark with the criterion crate rather than the nightly-only #[bench] attribute?

level: juniorimportance: should knowfreq 30%

answer

  1. nightly gate vs a dev-dependency
  2. benches/ directory, one file per target
  3. a function taking &mut Criterion
  4. bench_function, then b.iter
  5. criterion_group! and criterion_main!

basics

~10 s

Add criterion as a dev-dependency, write a function taking &mut Criterion in a benches/ file that calls bench_function with b.iter, and register it with criterion_group! and criterion_main!. The built-in #[bench] needs nightly's #![feature(test)].

solid answer

~40 s

On stable, the built-in benchmark support is closed: `#![feature(test)]` fails with E0554 and a bare `#[bench]` fails with E0658, because `#[bench]` and `test::Bencher` are unstable. The criterion crate replaces them. Add `criterion = "0.7"` under `[dev-dependencies]`, create `benches/checksum.rs`, and write `fn bench_checksums(c: &mut Criterion)` that calls `c.bench_function("fnv1a 64KiB", |b| b.iter(|| fnv1a(black_box(&buf))))`. Build the 64 KiB buffer before that call, because only the closure handed to `b.iter` runs inside the timing loop. `criterion_group!(benches, bench_checksums)` defines a function that runs the targets, and `criterion_main!(benches)` generates the `main`, so that bench target must switch off the default libtest harness in the Cargo manifest. Run it with Cargo's bench command, which compiles with optimisations.

code

rust · 15 lines
rust
use std::hint::black_box;

use checksums::{adler32, fnv1a};
use criterion::{Criterion, criterion_group, criterion_main};

fn bench_checksums(c: &mut Criterion) {
    // Setup runs once, outside the timed closure.
    let buf: Vec<u8> = (0..64 * 1024).map(|i| (i % 251) as u8).collect();

    c.bench_function("adler32 64KiB", |b| b.iter(|| adler32(black_box(&buf))));
    c.bench_function("fnv1a 64KiB", |b| b.iter(|| fnv1a(black_box(&buf))));
}

criterion_group!(benches, bench_checksums);
criterion_main!(benches);

go deeper

for a junior

Recall the pieces: criterion as a dev-dependency, a file in benches/, a function taking &mut Criterion, bench_function with b.iter, and the two macros. Know that #[bench] is nightly-only.

for a middle

Explain that only the closure given to b.iter is timed, why setup goes before it, and why the target needs the libtest harness switched off because criterion_main! supplies main.

for a senior

Show you have been burned: a target that silently ran zero benchmarks, numbers from a dev-profile build, a deprecated criterion::black_box left in old code. Say how you would notice each one.

for a principal

Frame criterion as a team default: where benchmarks live, which functions deserve one, and that a benchmark nobody runs in a release build is decoration rather than evidence.

## Why not the built-in `#[bench]` Rust's test harness, the `test` crate that runs `#[test]` functions, also contains a small benchmark runner: a function marked `#[bench]` takes a `&mut test::Bencher` and calls `b.iter(...)` with the code to time. That runner is **unstable**. On a stable 1.99 compiler both ways in fail: - `#![feature(test)]` at the crate root is rejected with **E0554**: "`#![feature]` may not be used on the stable release channel". - A bare `#[bench]` without the gate is rejected with **E0658**: "use of unstable library feature `test`: `bench` is a part of custom test frameworks which are unstable". So on stable you reach for a library. **criterion** (version 0.7 at the time of writing) is the usual choice: it runs on stable, warms the code up, takes many samples, computes confidence intervals and compares each run with the previous one. ## The five pieces of a criterion benchmark 1. **A dev-dependency.** `criterion = "0.7"` under `[dev-dependencies]`, so it is built for tests and benchmarks but never shipped with the library. 2. **A file in `benches/`.** Each file there is a separate bench target, compiled as its own crate that links your library, so it can only call the library's public items. 3. **A benchmark function** with the signature `fn name(c: &mut Criterion)`. Inside it, `c.bench_function("id", |b| b.iter(|| work()))` registers one benchmark. The `Bencher` that `b` refers to decides how many times to call the closure; you only say what one iteration is. 4. **Two macros.** `criterion_group!(benches, bench_checksums)` generates a function named `benches` that builds a `Criterion` with default settings, applies the command-line options and calls each listed function. `criterion_main!(benches)` generates `fn main()` that calls the groups and prints a final summary. 5. **The harness switched off for that target.** Because criterion brings its own `main`, the bench target must not use the default libtest harness. That is one key in Cargo's `[[bench]]` table, covered with Cargo's targets. ## What goes inside the closure, and what does not Only the closure passed to `b.iter` is repeated in the timing loop. Anything written before `bench_function` (building a 64 KiB buffer, reading a fixture file, constructing a parser) runs once and is not timed. Put setup inside the closure and you time the allocation together with the checksum. Inputs are wrapped in `std::hint::black_box`, a standard-library hint that keeps the optimiser from treating the buffer as a known constant. The return value of the closure is passed through `black_box` by criterion itself. ## What a run prints Each benchmark warms up for 3 seconds, then collects 100 samples over about 5 seconds; those are criterion's defaults. A result line shows three numbers in brackets: the lower bound of a 95% confidence interval, the point estimate and the upper bound. Criterion also counts outliers among the samples. On a second run it compares against the results saved by the first. ## Running one benchmark, or all of them Cargo builds every bench target and runs each binary; everything after Cargo's `--` separator is handed to those binaries (the command line itself is Cargo's subject). Criterion's binary understands its own options: - A positional **filter** is a regular expression: only benchmarks whose names match it run, and `--exact` demands the exact name instead. - `--list` prints the benchmark names without running them. - `--warm-up-time`, `--measurement-time` and `--sample-size` change the defaults for one run. One trap sits at this border: the options reach every bench binary Cargo runs, including a library's own libtest unit-test binary. When a criterion-only option such as `--quick` arrived there, libtest stopped with `error: Unrecognized option: 'quick'` before criterion ever ran, so pass the criterion target's name to Cargo as well. ## Mistakes that produce numbers you should not trust - **Leaving the default harness on.** The target still compiles, but libtest generates its own `main`, finds no `#[test]` or `#[bench]` functions and prints `running 0 tests` with `test result: ok`. Nothing warns you. - **Measuring an unoptimised build.** Cargo's bench command uses its bench profile, which inherits the release settings. Forcing the dev profile on one machine made the same 64 KiB Adler-32 benchmark about three times slower, and the slowdown differed between the two checksum functions. - **Using `criterion::black_box`.** Criterion 0.6 deprecated it; it now forwards to `std::hint::black_box` and triggers a `deprecated` warning. - **Expecting `#[bench]` on stable.** It stays nightly-only; a team on stable uses criterion or another crate. | | nightly `#[bench]` | criterion on stable | |---|---|---| | Toolchain | nightly, `#![feature(test)]` | any stable release criterion supports | | Where it lives | in the crate, beside tests | `benches/*.rs`, one target per file | | Output | one `ns/iter (+/- n)` line | interval, outliers, change against the last run | | Saved history | none | per-benchmark results under `target/criterion` |

  • What happens if the criterion bench target keeps the default libtest harness?
    It still compiles, but libtest generates its own `main`, so the one from `criterion_main!` never runs. The harness finds no `#[test]` or `#[bench]` functions and prints `running 0 tests` and `test result: ok`, with no warning. The fix is the `harness = false` key on that target in Cargo's manifest.
  • How does the output of nightly's #[bench] differ from criterion's?
    libtest prints one line per benchmark, such as `1 ns/iter (+/- 0)`. The number is the median of 50 winsorised samples, and the figure after `+/-` is their maximum minus their minimum, not a confidence interval. Nothing is saved between runs, so there is no comparison. Criterion reports an interval, classifies outliers and compares each run with the saved previous one.
  • Why does a criterion benchmark built in Cargo's dev profile mislead?
    It measures code without optimisation: little inlining, overflow checks on arithmetic, debug assertions on. On one machine the 64 KiB Adler-32 benchmark took about 415 µs in the dev profile against about 142 µs in the bench profile, and the slowdown was different for FNV-1a. A ranking from a debug build need not hold in the release build users run.

  • #[bench] works on stable as long as you import test::Bencher.
  • Building the input inside b.iter is fine because criterion subtracts setup time.
  • criterion_main! works under the default harness, so harness = false is optional.
  • A benchmark from a debug build is good enough to compare two functions.
  • criterion::black_box is the function criterion benchmarks should use.