In Catch2 3, how do you write a BENCHMARK inside a test case, and why should the benchmarked block return its result?
answer
- a lambda, so end with ;
- probe, estimate, measure
- returning keeps the work alive
- advanced form takes a Chronometer
basics
~20 sInclude catch2/benchmark/catch_benchmark.hpp and write BENCHMARK("name") { return compute(); }; inside a TEST_CASE. The block is a lambda, so it needs a semicolon, and returning the result is the documented way to keep the compiler from optimising the work away.
solid answer
~40 s`BENCHMARK("fib 20") { return fib(20); };` expands to a lambda that Catch2 runs many times, so the closing brace needs a semicolon. It lives inside an ordinary `TEST_CASE`, next to the assertions, and in Catch2 3 it comes from `catch2/benchmark/catch_benchmark.hpp`. Catch2 first probes the clock, then runs the code a few times to estimate how many runs fit in one sample, then collects the samples, 100 by default, and bootstraps a mean and standard deviation with confidence bounds. The code must therefore be repeatable. Returning the result is the documentation's way to stop the optimiser from discarding unused work. For set-up that must not be timed, `BENCHMARK_ADVANCED` passes a `Catch::Benchmark::Chronometer`, and only the callable given to `meter.measure` is timed. `--skip-benchmarks` skips the blocks, while a `[!benchmark]` tag hides a whole case.
code
cpp · 24 lines#include <catch2/catch_test_macros.hpp>
#include <catch2/benchmark/catch_benchmark.hpp>
#include <algorithm>
#include <cstdint>
#include <vector>
static std::uint64_t fib(std::uint64_t n) { return n < 2 ? 1 : fib(n - 1) + fib(n - 2); }
TEST_CASE("fibonacci", "[fib]") {
CHECK(fib(5) == 8);
BENCHMARK("fib 20") { return fib(20); };
}
TEST_CASE("sorting", "[sort]") {
std::vector<int> v(10000);
for (int i = 0; i < 10000; ++i) v[i] = (i * 7919) % 10007;
BENCHMARK_ADVANCED("sort 10k")(Catch::Benchmark::Chronometer meter) {
meter.measure([&] {
auto copy = v;
std::sort(copy.begin(), copy.end());
return copy.size();
});
};
}go deeper
Recall the BENCHMARK syntax inside a test case, its header, and the trailing semicolon.
Explain probe, estimation and measurement, why the code must be repeatable, and why the block returns its result.
Use BENCHMARK_ADVANCED to keep set-up out of the timing, recognise results that are too fast to be real, and separate benchmark runs from correctness runs in CI.
Decide whether micro-benchmarks belong in the unit-test binaries or in a dedicated harness, weighing convenience against noise, run time and the discipline benchmark results need.
## Benchmarks next to the tests Catch2 can time code inside an ordinary test case. In Catch2 3 the macros come from `catch2/benchmark/catch_benchmark.hpp`: ```cpp TEST_CASE("fibonacci", "[fib]") { CHECK(fib(5) == 8); BENCHMARK("fib 20") { return fib(20); }; } ``` Points of syntax and placement: - `BENCHMARK` expands to a **lambda**, so a semicolon must follow the closing brace. - It can appear anywhere inside a test case, so a check that the function is correct can precede the measurement of how fast it is. - The benchmark runs during a normal run of the binary; in a 3.16.1 run, the timing table appeared between the assertion and the next line of the test case. How to design a sound benchmark, from warm-up to noise control and profiling, is general performance work; this question is about what Catch2's macros do. ## How Catch2 executes a benchmark The documentation describes three steps: 1. **Environmental probe**: before the first benchmark, Catch2 estimates the clock's resolution and the cost of calling it. 2. **Estimation**: the user code is run a few times to decide how many runs make up one **sample**, so each sample lasts long enough for the clock to measure accurately. 3. **Measurement**: the samples are collected one after another, each with that number of runs. Because the number of runs is decided at run time, the code **must be repeatable**: code that consumes a one-shot resource or mutates state it depends on gives bogus results or crashes. After measurement, Catch2 bootstraps the samples to report a mean and standard deviation with lower and upper bounds. | Option | Default | Effect | |---|---|---| | `--benchmark-samples` | 100 | number of samples | | `--benchmark-resamples` | 100000 | bootstrap resamples | | `--benchmark-confidence-interval` | 0.95 | confidence for the bounds | | `--benchmark-no-analysis` | off | report only the plain mean | | `--skip-benchmarks` | off | skip `BENCHMARK` blocks, added in 3.0.1 | ## Why return the result If the computed value is never used, the optimiser may delete the computation, and the benchmark then times an empty lambda. The documentation calls returning the value a handy way to avoid that, because Catch2 consumes what the lambda returns. Returning is not a guarantee against every optimisation: a call whose arguments are compile-time constants can be folded before the benchmark ever runs. A 3.16.1 run of `fib(20)` built with `-O2` reported well under a nanosecond per call, which is the signature of a result computed at compile time rather than measured. ## BENCHMARK_ADVANCED for untimed set-up The simple form times its whole body. When set-up must be excluded, the advanced form receives a `Catch::Benchmark::Chronometer`: ```cpp BENCHMARK_ADVANCED("sort 10k")(Catch::Benchmark::Chronometer meter) { std::vector<int> v = shuffled_ints(10000); // the test's own helper meter.measure([&] { auto copy = v; std::sort(copy.begin(), copy.end()); return copy.size(); }); }; ``` - Only the callable passed to `meter.measure` is timed. - The advanced block itself runs exactly twice, once for estimation and once for measurement, while the callable runs as many times as needed. - A callable that takes an `int` receives the run index, and `meter.runs()` gives the number of runs, so each run can mutate its own prepared copy. - `BENCHMARK("name", i) { ... };` is a shorthand for the indexed form. ## Reading the console output The console reporter prints a block per benchmark with three rows of columns: - The first row holds the **benchmark name**, the number of **samples**, the **iterations** (runs per sample) chosen during estimation, and the **estimated run time** for the whole measurement. - The second row is the **mean** with its **low** and **high** bootstrap bounds. - The third row is the **standard deviation** with its bounds. A wide gap between the low and high means, or a standard deviation close to the mean, signals a noisy measurement rather than a property of the code. ## Benchmarks and test runs - `--skip-benchmarks` skips the `BENCHMARK` and `BENCHMARK_ADVANCED` blocks but still runs the rest of the test case, assertions included. - A test case tagged `[!benchmark]` is hidden from the default run, like `[.]`, and runs only when selected by name or tag. - Keeping benchmarks in separate, tagged test cases lets CI run correctness tests on every change and benchmarks on a schedule.
- In Catch2, what is the difference between --skip-benchmarks and tagging a test case [!benchmark]?`--skip-benchmarks`, added in 3.0.1, skips the `BENCHMARK` and `BENCHMARK_ADVANCED` blocks at run time but still runs the rest of each test case, assertions included. A `[!benchmark]` tag marks the whole test case as a benchmark, which hides it from the default run until a spec selects it by name or tag.
- Why must the code inside a Catch2 BENCHMARK be safe to run many times?Catch2 decides the number of runs at run time: it executes the code a few times during estimation, then repeatedly for each of the samples. Code that pops from a shared queue, moves from an object or appends to a container it then measures changes its own input between runs, which skews the timing or crashes. `BENCHMARK_ADVANCED` with a run index lets each run use its own prepared copy.
- A Catch2 BENCHMARK body is a plain block, so no semicolon is needed.
- BENCHMARK runs its body exactly once and reports that time.
- --skip-benchmarks also skips the assertions in the same test case.
- Returning a value from BENCHMARK guarantees the compiler cannot precompute it.
- BENCHMARK_ADVANCED times everything inside its block.
Assumes Catch2 3, where benchmarks come from catch2/benchmark/catch_benchmark.hpp and --skip-benchmarks exists since 3.0.1; checked against 3.16.1.