In a Rust criterion benchmark of seal(frame: Vec<u8>), which consumes a 64 KiB buffer, why use iter_batched, and why did BatchSize::SmallInput use 3 GB of memory?
answer
- the routine eats its input
- untimed setup closure
- outputs dropped after the clock
- a tenth of the iterations at once
- warm-up keeps doubling
basics
~20 sseal consumes its buffer, so each iteration needs a fresh copy, which iter_batched builds in untimed setup. SmallInput builds a tenth of each timing call's iterations at once, and warm-up keeps doubling that count, so 64 KiB inputs pile up.
solid answer
~40 sWith `b.iter(|| seal(payload.clone()))` the timed loop includes the clone, the seal and dropping the returned frame. `b.iter_batched(|| payload.clone(), seal, BatchSize::SmallInput)` builds inputs in an untimed setup closure, times only `seal`, and drops the outputs after the clock stops. Here the clone cost about 0.65 µs of a 17.7 µs iteration, so both forms agreed within noise; the gain grows as setup approaches the routine's cost. The memory comes from batching: `SmallInput` makes each batch `ceil(iters / 10)` inputs, all built before the batch is timed. Warm-up calls the routine with 1, 2, 4… iterations until 3 seconds pass, so its last round held over 13,000 copies, and peak memory reached 3.2 GB. `LargeInput` (`ceil(iters / 1000)`) gave the same time at 66 MB.
code
rust · 23 linesuse checksums::seal;
use criterion::{BatchSize, Criterion, criterion_group, criterion_main};
fn bench_seal(c: &mut Criterion) {
let payload: Vec<u8> = (0..64 * 1024).map(|i| (i % 251) as u8).collect();
let mut group = c.benchmark_group("seal-64KiB");
// Times the clone, the seal and the drop of the returned frame.
group.bench_function("iter", |b| b.iter(|| seal(payload.clone())));
// Clones in an untimed setup; outputs are dropped after the clock stops.
for (name, size) in [
("small", BatchSize::SmallInput),
("large", BatchSize::LargeInput),
("per-iteration", BatchSize::PerIteration),
] {
group.bench_function(name, |b| b.iter_batched(|| payload.clone(), seal, size));
}
group.finish();
}
criterion_group!(benches, bench_seal);
criterion_main!(benches);go deeper
Know that iter_batched exists for routines that consume their input, and that its setup closure is not timed.
Explain the steps of one batch, why outputs are dropped after the clock stops, and how BatchSize sets the number of inputs built at once.
Diagnose the 3 GB run: the batch formula, the warm-up's doubling, the input size. Pick LargeInput, and show that the timing did not change.
Set the expectation that benchmark reviews look at memory and setup cost as well as the headline time, and that a loop choice is justified by a measurement.
## Why `b.iter` is the wrong loop for a consuming routine `seal(frame: Vec<u8>) -> Vec<u8>` takes ownership of a frame, appends a four-byte Adler-32 trailer and returns it. Because it **consumes** its argument, one prepared buffer cannot be reused across iterations; every call needs its own copy. With criterion's plain loop that copy has to be made inside the timed closure: `b.iter(|| seal(payload.clone()))` Criterion's documented timing model for `iter` is the routine plus dropping its output, on every iteration. Here the "routine" is clone, seal and the drop of the returned frame. Whether that matters depends on proportions: on the test machine a 64 KiB clone took about 0.65 µs and the whole iteration about 17.7 µs, so the two forms agreed within noise. For a cheaper routine, or more expensive setup, the clone would dominate. ## What `iter_batched` times `b.iter_batched(setup, routine, size)` separates the two. In criterion 0.7 each timing call does this: 1. Compute a batch size from the `BatchSize` value and the number of iterations requested. 2. Call `setup` once per iteration in the batch and **collect all the inputs** into a `Vec`, before the clock starts. 3. Start the clock, feed every input to `routine`, collect the outputs, stop the clock. 4. Pass the outputs to `black_box` and drop them, **after** the clock stopped. So setup and the output's destructor are not measured. A routine that drops its own input inside its body still pays for that drop; `seal` returns its buffer, so it does not. For a routine that takes `&mut I` (sorting in place, scrubbing a buffer), `iter_batched_ref` hands each input by mutable reference and drops both input and output outside the timed region. ## How `BatchSize` trades memory for overhead | Variant | Inputs per batch | Overhead the docs quote | Peak memory measured | |---|---|---|---| | `SmallInput` | `ceil(iters / 10)` | about 500 ps | 3.2 GB | | `LargeInput` | `ceil(iters / 1000)` | about 750 ps | 66 MB | | `PerIteration` | 1 | about 350 ns | 60 MB | | `NumBatches(n)` | `ceil(iters / n)` | you tune it | not measured | | `NumIterations(n)` | `n` | you tune it | not measured | Plain `iter` peaked at 59 MB in the same experiment. All four timings fell between 17.5 and 17.8 µs. The memory figures are the whole bench process's peak resident memory on one laptop; the overhead figures are criterion's own rough guide, which its docs say must not be subtracted from a measurement. ## The warm-up multiplies the problem The sampling phase was modest: about 270,000 iterations spread over 100 samples, so the largest sample asked for under 6,000 iterations. The warm-up is different. Criterion calls the routine with 1, then 2, then 4 iterations, doubling until the elapsed time passes the 3-second warm-up. With `iter_batched` that elapsed time includes setup, and with a debug trace the warm-up completed 262,143 iterations, so its last call asked for 131,072. A tenth of that is over 13,000 inputs of 64 KiB held at once, plus the outputs, each grown to fit the trailer. Shortening the warm-up confirmed the cause: with `--warm-up-time 0.1` peak memory fell to 153 MB, at 1 second it was 540 MB, and at the default 3 seconds it was 2.6 GB on that run. ## Choosing a loop - Use **`iter`** when there is no per-iteration setup, or when setup is negligible next to the routine and you have checked that it is. - Use **`iter_batched`** when the routine consumes its input; start with **`SmallInput`**, which criterion recommends for small values. - Switch to **`LargeInput`** when each input is large, as a 64 KiB buffer is; it was just as accurate here. - Use **`PerIteration`** only for inputs that hold a scarce resource, such as a file handle, since its overhead is in nanoseconds. - Use **`iter_batched_ref`** when the routine mutates its input in place. - Watch memory, not just time, when a benchmark's inputs are large. The general lesson is that a benchmark harness is itself a program with costs. Criterion hides setup and teardown from the clock by doing them in bulk, and bulk means memory. On a laptop with plenty of RAM the 3 GB run finished and reported a perfectly good number; on a CI runner with less memory the same benchmark could be killed or start swapping, and a swapping benchmark measures the disk.
- When would you use iter_batched_ref instead of iter_batched?When the routine works on its input in place and takes `&mut I`, such as sorting a vector or zeroing a buffer. Setup still builds a fresh input per iteration outside the clock, the routine gets a mutable reference, and criterion drops both the input and the output after timing stops. With `iter_batched` such a routine would need a wrapper that takes ownership.
- Why not time the clone separately and subtract it from the iter result?Two noisy estimates subtracted carry both uncertainties, and a clone inside the loop does not cost what it costs alone, because cache and allocator state differ. Criterion's own `BatchSize` docs say it is not valid to subtract even its quoted overheads. Keeping setup out of the timed region is the reliable way.
- How did you show that the warm-up, not the sampling, drove the memory?By changing only the warm-up time. With `--warm-up-time 0.1` the `SmallInput` run peaked at 153 MB, with 1 second at 540 MB and with the default 3 seconds at 2.6 GB, while the timings barely moved. Criterion's debug trace showed the warm-up had completed 262,143 iterations, so its last doubling asked for 131,072.
- iter_batched times the setup closure as well, just in batches.
- SmallInput is always the right choice because it has the lowest overhead.
- PerIteration is the safest default since it never holds more than one input.
- Memory use of a benchmark only depends on the sample count, not on warm-up.
- Subtracting a separately measured clone time gives the routine's true cost.