skip to content

Generators and Library Setup

GENERATE runs a case once per value, BENCHMARK times code, and version 3 ships as a compiled library rather than v2's single header. Interviewers ask how to wire it into CMake.

on this pageshow

explore

questions

6

How do you run only some Catch2 tests from the command line and write JUnit XML for CI in the same run?

level: juniorimportance: should knowfreq 30%

answer

  1. test specs: names, wildcards, tags
  2. a tilde excludes
  3. reporters take options after ::
  4. several reporters in one run

basics

~20 s

Pass a test spec, such as a name, "area*" or "[geometry]~[slow]", to the test binary, and choose reporters with -r. Since Catch2 3.0.1, several can run at once: -r console -r junit::out=results.xml writes readable output and a JUnit file.

solid answer

~50 s

Arguments that are not options form a **test spec**. A full name selects one case, a name with `*` at the start or end is a wildcard, `[tag]` selects by tag, adjacent specs must all match, a comma means or, and `~` excludes; hidden cases run only when a spec selects them. `-c "section name"` narrows a run to one section, and `--list-tests` shows what a spec matches. Output format is the **reporter**: `-r junit` writes JUnit-style XML, and options follow `::`, so `-r junit::out=results.xml` writes to a file. Since Catch2 3.0.1, `-r` can be given more than once, so `-r console -r junit::out=results.xml` gives human output and a CI file from one run. The binary exits with 0 when all tests pass and 42 when any fail. Since 3.9.0 the default order is random, so pass `--order decl` or a fixed `--rng-seed` to reproduce an order.

code

bash · 8 lines
bash
# What does the spec match?
./tests "[geometry]~[slow]" --list-tests

# Run it, with readable output and a JUnit file for CI
./tests "[geometry]~[slow]" -r console -r junit::out=results.xml

# Only one section of one test case, in declaration order
./tests "generators inside sections" -c "two" --order decl

go deeper

for a junior

Recall how to select tests by name, wildcard and tag, and how -r junit::out=file writes a JUnit report.

for a middle

Explain how specs combine, why hidden cases need an explicit spec, how several reporters share one run, and what the exit codes signal.

for a senior

Build CI invocations that are reproducible and fail loudly: fixed seeds for random order, specs that cannot silently match nothing, sharding and per-job reports.

for a principal

Standardise the test-binary interface across a codebase's suites, so every CI job selects, orders and reports tests the same way and failures reproduce locally.

## The test binary is the runner A Catch2 test executable carries its own command-line interface, whether the `main` comes from `Catch2WithMain` or from your own `Catch::Session`. Everything below works the same way when the binary runs under CTest, because CTest simply passes arguments to it. ## Selecting tests with test specs Positional arguments form a **test spec**. Without one, the binary runs every non-hidden case. The basic specs: - **A full name**, such as `"area is positive"`, selects that case. - **A wildcarded name**, with `*` only at the start or end, such as `"area*"` or `"*triangle*"`. - **A tag**, such as `[geometry]`. They combine: 1. Specs written next to each other must **all** match: `[geometry][fast]`. 2. A **comma** means any: `[geometry],[migration]`. Commas bind more loosely than concatenation, so `[a][b],[c]` means both a and b, or c. 3. A **tilde** rejects: `[geometry]~[slow]`. Rejection wins over everything else, and `~` binds only to the next basic spec. A hidden case, tagged `[.]`, `[.name]` or `[!benchmark]`, runs only when a positive spec selects it. Quote specs in the shell so `*` and brackets reach the binary unchanged, and use `--list-tests` with the same spec to see what it matches before running anything. ## Narrowing further - `-c "section name"` runs only that section path; repeat `-c` for nested sections. - Catch2 3.13.0 added `-g` and `-p` for generator elements and general path filters. - `-#` adds a `[#filename]` tag to every case, so `"[#test_area]"` selects one source file's cases. - `--shard-count` and `--shard-index` split the cases across parallel CI jobs. ## Listing and sharding - `--list-tests` prints the matching cases with their tags, and `--list-tags` prints every tag with a count; both accept the same test spec as a run. - `--shard-count N --shard-index i`, added in Catch2 3.0.1, splits the selected cases evenly into `N` sets and runs set `i`, counting from 0, so `N` parallel CI jobs can each run one share of a large suite. - Sharding happens after filtering, so every job must pass the same spec and the same seed for the shares to cover the suite exactly once. ## Choosing output with reporters The **reporter** formats results. `--list-reporters` printed these for 3.16.1: `Automake`, `compact`, `console` (the default), `JSON`, `JUnit`, `SonarQube`, `TAP`, `TeamCity` and `XML`. | Need | Arguments | |---|---| | Readable local output | default `console`, `-s` to show passing assertions too | | JUnit XML file for CI | `-r junit::out=results.xml` | | Both from one run | `-r console -r junit::out=results.xml` | | Quiet console | `-r console::verbosity=quiet` | Reporter options follow `::` as `key=value` pairs. Running **several reporters** at once was introduced in Catch2 3.0.1; before that, one run produced one report format. Each reporter needs its own `-r`: a comma-joined `-r console,junit` was rejected with `Unrecognized reporter, 'console,junit'`. A run with `-r console -r junit::out=res.xml` printed the console summary and wrote a `<testsuites>` document. How a CI server reads that JUnit file is a separate subject. ## Order, seeds and exit codes - **Order.** Since Catch2 3.9.0 the default `--order` is `rand`; `decl` (declaration order) and `lex` (by name) are the alternatives. A suite that only passes in one order has hidden coupling, which a random default exposes. - **Seed.** The seed is printed at the start of each run; `--rng-seed <n>` reproduces both the order and the random generator values. - **Exit code.** The `Catch::Session` constants in 3.16.1 include `TestFailureExitCode` = 42 and `NoTestsRunExitCode` = 2. A spec that matched nothing printed `No test cases matched` and exited with 2, so a CI job with a typo in its spec fails instead of passing vacuously. ## A typical CI line ```bash ./tests "[geometry]~[slow]" --order rand --rng-seed 1234 \ -r console::verbosity=quiet -r junit::out=results.xml ``` This runs the fast geometry cases in a reproducible random order, keeps the log short, and leaves a JUnit file for the CI server.

  • Why might a Catch2 suite start failing after an upgrade to 3.9.0 without any test changes?
    Catch2 3.9.0 changed the default run order to random. Tests that silently depended on running after another test, through a global, a static cache or a file, now meet a different order and fail intermittently. Rerunning with the printed `--rng-seed` reproduces the order; the fix is removing the shared state, with `--order decl` only as a temporary bridge.
  • What exit code does a Catch2 binary return when its test spec matches no test cases?
    It prints `No test cases matched` and exits with 2, the `NoTestsRunExitCode` constant in `Catch::Session`. That makes a mistyped filter in a CI script fail visibly. A test failure exits with 42 (`TestFailureExitCode`), and `--allow-running-no-tests` exists for jobs where an empty selection is legitimate.

  • A space between two tags in a Catch2 spec means either tag.
  • Catch2 can only write one report format per run.
  • Catch2 runs tests in declaration order unless told otherwise.
  • A failing Catch2 run exits with 1.
  • Hidden tests run whenever no filter is given.

Assumes Catch2 3: several reporters per run since 3.0.1, random default order since 3.9.0, generator and path filters since 3.13.0; checked against 3.16.1.

open as a page

In Catch2, how does GENERATE run one TEST_CASE once per value, and what do the take, filter and range adaptors do?

level: middleimportance: should knowfreq 25%

basics

~20 s

GENERATE(1, 3, 5) makes Catch2 enter the test case once per value, and several GENERATE calls give their cartesian product. Adaptors build sequences: range(a, b) stops before b, filter keeps matching values, take keeps the first n.

open as a page

How does Catch2's catch_discover_tests register test cases with CTest, and why can it fail the build or silently miss tests?

level: seniorimportance: should knowfreq 30%

basics

~20 s

catch_discover_tests, from Catch2's Catch.cmake, runs the built binary with --list-tests and adds one CTest test per TEST_CASE. Listing runs after the build by default, so a binary that cannot list fails the build; hidden cases need a TEST_SPEC.

open as a page

When migrating a CMake project from Catch2 v2's single catch.hpp to Catch2 v3, what changes in the includes, the main function and the link targets?

level: seniorimportance: should knowfreq 30%

basics

~10 s

Catch2 v3 is a compiled static library with per-feature headers. Replace catch.hpp with catch_test_macros.hpp and the feature headers you use, delete the CATCH_CONFIG_MAIN file, and link Catch2::Catch2WithMain, or Catch2::Catch2 plus your own main.

open as a page

In Catch2 3, how do you write a BENCHMARK inside a test case, and why should the benchmarked block return its result?

level: middleimportance: nice to knowfreq 15%

basics

~20 s

Include catch2/benchmark/catch_benchmark.hpp and write BENCHMARK("name") { return compute(); }; inside a TEST_CASE. The block is a lambda, so it needs a semicolon, and returning the result is the documented way to keep the compiler from optimising the work away.

open as a page

A Catch2 test calls GENERATE(1, 2), then has SECTION one with GENERATE(-3, -2) and SECTION two with GENERATE(4, 5, 6). How many passes run, and why?

level: seniorimportance: nice to knowfreq 15%

basics

~20 s

Ten passes. GENERATE acts as an implicit section reaching to the end of its scope, so each outer value replays every leaf below it: section one runs 2 x 2 = 4 times, section two 2 x 3 = 6.

open as a page