skip to content

How does Catch2's catch_discover_tests register test cases with CTest, and why can it fail the build or silently miss tests?

level: seniorimportance: should knowfreq 30%

answer

  1. it runs the binary to list tests
  2. after the build, by default
  3. one CTest test per TEST_CASE
  4. hidden cases are not listed

basics

~20 s

catch_discover_tests, from Catch2's Catch.cmake, runs the built binary with --list-tests and adds one CTest test per TEST_CASE. Listing runs after the build by default, so a binary that cannot list fails the build; hidden cases need a TEST_SPEC.

solid answer

~40 s

After `find_package(Catch2 3 REQUIRED)`, `include(Catch)` provides `catch_discover_tests(target)`. It does not parse source code: it **runs the test executable** with `--list-tests`, which in 3.16.1's `CatchAddTests.cmake` uses the JSON reporter, and registers **one CTest test per `TEST_CASE`**, each invoking the binary with that case's name. Sections and generator values are not separate CTest tests. By default discovery happens as a post-build step, so if the binary cannot list its tests, for example because of a duplicate name and tags, the **build** fails with `Error listing tests from executable`. Discovery uses the default test spec, so hidden cases such as `[.integration]` are not registered unless a `TEST_SPEC` selects them, ideally with a `TEST_PREFIX`. When the binary cannot run on the build machine, `DISCOVERY_MODE PRE_TEST` defers listing until `ctest` runs.

code

cmake · 12 lines
cmake
cmake_minimum_required(VERSION 3.16)
project(shapes_tests LANGUAGES CXX)

find_package(Catch2 3 REQUIRED)

add_executable(tests test_area.cpp)
target_link_libraries(tests PRIVATE Catch2::Catch2WithMain)

include(CTest)
include(Catch)
catch_discover_tests(tests)
catch_discover_tests(tests TEST_SPEC "[integration]" TEST_PREFIX "integration: ")

go deeper

for a junior

Recall that catch_discover_tests comes from include(Catch) and registers each TEST_CASE with CTest so ctest can run them.

for a middle

Explain that discovery runs the binary to list test cases, when it runs, and why sections and generator values are not separate CTest tests.

for a senior

Diagnose discovery failures in the build, register hidden suites deliberately with TEST_SPEC and TEST_PREFIX, and switch to PRE_TEST when cross-compiling.

for a principal

Decide the granularity CTest should see for a project, balancing parallel runs and readable reports against the cost of splitting cases and of discovery steps in every build.

## Where the function comes from Catch2's repository ships CMake scripts in its `extras` directory, and an installed Catch2 3.16.1 puts them next to its package configuration: `Catch.cmake` with its helper `CatchAddTests.cmake`, plus sharding scripts and the deprecated `ParseAndAddCatchTests.cmake`. After `find_package(Catch2 3 REQUIRED)`, `include(Catch)` finds them. With `FetchContent` the documentation says `include(Catch)` fails unless `CMAKE_MODULE_PATH` is extended with the fetched `extras` directory. ```cmake find_package(Catch2 3 REQUIRED) add_executable(tests test_area.cpp) target_link_libraries(tests PRIVATE Catch2::Catch2WithMain) include(CTest) include(Catch) catch_discover_tests(tests) ``` CTest itself, `enable_testing`, `add_test` and running `ctest`, belongs to the CMake material; this function is the registration call Catch2 contributes. ## How discovery works 1. CMake adds a step that **runs the built executable** with `--list-tests`. In 3.16.1, `CatchAddTests.cmake` passes `--reporter json` and `--order lex` so the listing is stable. 2. The script parses the listing and writes a file of `add_test` calls that CTest reads. 3. Each registered test runs the binary with the test case's name as a filter. `ctest -V -N` showed the command `tests "area is positive" "--order decl"`. The listing contains **test cases**, so: - One `TEST_CASE` is one CTest test, however many sections or generator values it holds. - A `TEMPLATE_TEST_CASE` contributes one CTest test per type, because each type is its own test case. - A failing section shows up in CTest only as its whole test case failing. ## Why it can fail the build With the default `DISCOVERY_MODE POST_BUILD`, listing happens during the build. Anything that stops the binary from listing therefore fails the **build**, not a test. A binary with two cases declared with the same name and tags refuses to start, and the build stopped with: ```plaintext CMake Error at .../CatchAddTests.cmake:307 (message): Error listing tests from executable '.../build2/tests': Result: 1 ``` Other causes are the same in kind: a crash in a static initialiser, a missing shared library at build time, or a binary built for another architecture. For the last two, `DISCOVERY_MODE PRE_TEST` defers listing until `ctest` runs. The documentation also requires `PRE_TEST` for the Xcode generator on arm64 macOS, because the binary is signed only after the post-build step. `CMAKE_CATCH_DISCOVER_TESTS_DISCOVERY_MODE` sets the default for a whole project. ## Why it can miss tests - **Hidden cases.** Discovery lists what a plain run would run, so cases tagged `[.]` or `[.integration]` are not registered. A second call, `catch_discover_tests(tests TEST_SPEC "[integration]" TEST_PREFIX "integration: ")`, registered the hidden case as `integration: slow integration`. - **A narrowed spec.** A `TEST_SPEC` that is too specific drops cases without warning. ## Useful options | Option | Effect | |---|---| | `TEST_SPEC` | test spec passed when listing, such as tags | | `TEST_PREFIX`, `TEST_SUFFIX` | distinguish tests from several calls on one binary | | `EXTRA_ARGS` | extra command-line arguments for every test | | `REPORTER`, `OUTPUT_DIR` | per-test reporter and output file, avoiding races under `ctest -j` | | `ADD_TAGS_AS_LABELS` | turns Catch2 tags into CTest labels | | `SKIP_IS_FAILURE` | reports skipped Catch2 tests as failures | | `DISCOVERY_MODE` | `POST_BUILD` (default) or `PRE_TEST` | ## Names, labels and parallel runs - The CTest test name is the Catch2 test case name, plus any `TEST_PREFIX` or `TEST_SUFFIX`, so `ctest -R` filters by the same names a developer sees in Catch2's own output. - `ADD_TAGS_AS_LABELS` turns tags into CTest labels, so `ctest -L` can select by tag without a second discovery call. - Each discovered test runs in its own process, which lets `ctest -j` spread test cases across cores; global state no longer leaks between cases, but each process pays the binary's start-up cost. - A reporter writing files, such as a JUnit reporter, should use `OUTPUT_DIR` so parallel processes do not overwrite one output file. ## Choosing it over the alternatives - A single `add_test` running the whole binary is simple, but CTest then sees one test, so `ctest -j` cannot parallelise and a report shows one red line. - `ParseAndAddCatchTests` parses source files and is deprecated since Catch2 2.13.4; the documentation warns it silently ignores tests using macros it cannot parse. - `catch_discover_tests` asks the binary itself, so what CTest knows matches what the binary contains.

  • Why does a Catch2 test case with five sections appear as one test in ctest output?
    `catch_discover_tests` registers what `--list-tests` reports, and the listing contains test cases, not sections or generator values. CTest runs the binary with the test case's name, and Catch2 then executes every section inside that one process, so CTest sees a single pass or fail. Splitting into separate test cases is the way to give CTest finer granularity.
  • When should catch_discover_tests use DISCOVERY_MODE PRE_TEST?
    When the test binary cannot run on the build machine at build time: cross-compiling for another architecture, depending on a runtime library only present in the test environment, or using the Xcode generator on arm64 macOS, where the binary is signed only after the post-build step. `PRE_TEST` defers listing until `ctest` runs; the generated script lists again only when its saved list is missing or older than the test binary, so the cost falls on the first `ctest` run after each rebuild.

  • catch_discover_tests parses the source files for TEST_CASE macros.
  • Every SECTION becomes its own CTest test.
  • Hidden test cases are registered with CTest automatically.
  • A duplicate test name only fails at ctest time, never during the build.
  • include(Catch) works out of the box with FetchContent.