skip to content

Which ctest options run a CMake test suite in parallel, select tests by name or label, and show the output of failures?

level: middleimportance: must knowfreq 55%

answer

  1. -j for parallel
  2. -R and -E match names
  3. -L and -LE match labels
  4. regexes, not exact names
  5. --rerun-failed reads the last run

basics

~20 s

ctest -j N runs up to N tests at once; -R and -E select or exclude tests by name regex, -L and -LE by label regex; --output-on-failure prints failing tests' captured output; --rerun-failed repeats the last run's failures.

solid answer

~40 s

`ctest -j 8` (or `--parallel`) runs up to eight tests concurrently, and the `CTEST_PARALLEL_LEVEL` environment variable sets the same thing for CI; since CMake 3.29 a bare `-j` uses the processor count, or 2 if that is larger. Selection uses **regular expressions**: `-R parser` runs tests whose names contain `parser`, `-E slow` excludes, `-L unit` runs tests whose `LABELS` property matches, `-LE gpu` excludes them, and several `-L` options must all match. `--output-on-failure` (or `CTEST_OUTPUT_ON_FAILURE=1`) prints captured output only for failing tests, `-N` lists what would run, `--rerun-failed` repeats only the last run's failures, and `--output-junit results.xml` (3.21+) writes a report for CI. A test preset can store these as `execution.jobs`, `filter.include.label` and `output.outputOnFailure`.

code

console · 15 lines
console
$ ctest --test-dir build --timeout 2 -L unit
...
100% tests passed out of 1

Label Time Summary:
fast    =   0.11 sec*proc (1 test)
unit    =   0.11 sec*proc (1 test)
$ ctest --test-dir build --timeout 2
1/3 Test #1: slow .............................***Timeout   2.02 sec
2/3 Test #2: fails ............................***Failed    0.00 sec
3/3 Test #3: quick ............................   Passed    0.00 sec
$ ctest --test-dir build --rerun-failed --output-on-failure
1/2 Test #1: slow .............................   Passed    4.02 sec
2/2 Test #2: fails ............................***Failed    0.00 sec
done

go deeper

for a junior

Run tests with ctest -j, pick a subset with -R or -L, and add --output-on-failure to see why something failed.

for a middle

Explain that the filters are regexes, that several -L options combine with AND, what --rerun-failed ignores, and how fixtures pull in extra tests.

for a senior

Make CI runs deterministic and readable: anchored filters, --no-tests=error, JUnit output, and a parallel level that the tests' resource needs allow.

for a principal

Agree on a label vocabulary and keep the CI test invocation in a preset, so every pipeline selects and reports tests the same way.

## Parallel runs A repository with 300 test executables needs them to run concurrently in CI: - `ctest -j 8` or `ctest --parallel 8` runs up to eight tests at once. - Since CMake 3.29, `-j` without a number uses the number of processors, or 2 if that is larger, and `-j 0` means unbounded. - The `CTEST_PARALLEL_LEVEL` environment variable has the same effect as `--parallel` and is ignored when `-j` is given on the command line; since 3.29 an empty value means the default level. - `--test-load <level>` holds back new tests while the CPU load is above a threshold. Parallelism is safe only for tests that do not share files, ports or other state; the material on test properties covers `RESOURCE_LOCK` and fixtures for the ones that do. ## Selecting tests | Option | Selects | |---|---| | `-R <regex>` / `--tests-regex` | tests whose names match | | `-E <regex>` / `--exclude-regex` | removes tests whose names match | | `-L <regex>` / `--label-regex` | tests with a label that matches | | `-LE <regex>` / `--label-exclude` | removes tests with a matching label | | `-N` / `--show-only` | lists the selection without running it | | `--rerun-failed` | only the tests that failed in the previous run | Three details trip people up: 1. These are **regular expressions**, not exact names. `-R parser` also selects `parser_fuzz`; use `-R '^parser$'` for one test, and `-L '^unit$'` to avoid matching a label like `unit_slow`. 2. Several `-L` options combine with **AND**: `-L unit -L fast` runs tests that carry labels matching both. 3. `--rerun-failed` ignores `-R`, `-E`, `-L` and `-LE`, using the failure list from the last run instead. In a CMake 4.4.4 run it re-ran exactly the two failed tests from the previous invocation. When a selected test requires a **fixture**, `ctest` adds that fixture's setup and cleanup tests automatically, so `-R query_users` can also run `db_start` and `db_stop`. ## Seeing what failed CTest captures each test's standard output and error. By default the summary only names the failures: - `--output-on-failure` prints the captured output of failing tests only, which keeps CI logs readable; `CTEST_OUTPUT_ON_FAILURE=1` in the environment does the same. - `-V` prints the output of every test, and `-VV` adds more of CTest's own detail. - The full log is always written to `Testing/Temporary/LastTest.log` in the build tree. - `--output-junit results.xml` (CMake 3.21+) writes JUnit XML, which most CI systems display natively. After a failing run, CTest itself suggests the next command: `Use "--rerun-failed --output-on-failure" to re-run the failed cases verbosely.` ## Other everyday options - `--test-dir build` (CMake 3.20+) runs from anywhere. - `-C Release` picks the configuration in a multi-config build tree. - `--timeout 60` gives tests without their own `TIMEOUT` property a default limit. - `--stop-on-failure` stops at the first failure, useful for a quick local check. - `--no-tests=error` fails the run when the selection is empty, so a typo in `-L` cannot turn into a green CI job. ## A CI recipe for a large suite For the 300-executable repository, a merge-gate job typically looks like this: ```bash ctest --test-dir build -C Release -j 16 -LE '^(slow|gpu)$' \ --timeout 120 --no-tests=error \ --output-on-failure --output-junit junit.xml ``` - `-LE` keeps slow and hardware-bound groups out of the gate; a nightly job runs them with `-L`. - `--timeout` caps tests that never set their own limit. - `--no-tests=error` turns an empty selection into a failure; in a CMake 4.4.4 run, `-L nomatch` exited 0 without it and 8 with it. - The JUnit file lets the CI system show per-test history. ## Writing it down once Typing the same flags in every CI job invites drift. A **test preset** in `CMakePresets.json` stores them: `configurePreset` names the build tree, `configuration` the `-C` value, `output.outputOnFailure`, `execution.jobs`, `execution.timeout` and `filter.include.label` the rest, and `ctest --preset ci` applies them. The structure of presets files belongs to the material on presets.

  • How do you make a typo in a CI job's ctest -L argument fail the build instead of passing silently?
    Add `--no-tests=error`. When the label regex matches nothing, ctest otherwise runs zero tests and exits successfully in normal test mode. A test preset can set the same through `execution.noTestsAction`, and `ctest -N -L <regex>` shows in advance which tests a filter selects.
  • Why does ctest -R query_users also run tests named db_start and db_stop?
    `query_users` lists a fixture in `FIXTURES_REQUIRED`, and `db_start` and `db_stop` are that fixture's `FIXTURES_SETUP` and `FIXTURES_CLEANUP` tests. When a selection includes a test that needs a fixture, ctest adds the fixture's setup and cleanup tests automatically; `-FA`, `-FS` and `-FC` exclude them when you really want the test alone.

  • ctest -R takes an exact test name
  • Two -L options run tests that match either label
  • --rerun-failed can be narrowed further with -R
  • ctest -j is unsafe in general, so CI should always run tests serially
  • CTest hides failing test output unless you rerun the binary by hand