skip to content

A test in a CMake suite sometimes hangs in CI and fails about once in fifty runs; how do ctest timeouts and the --repeat modes help you contain and find it?

level: seniorimportance: nice to knowfreq 30%

answer

  1. TIMEOUT beats --timeout
  2. include(CTest) default 1500 s
  3. until-fail finds, until-pass hides
  4. after-timeout for busy machines
  5. --schedule-random for order bugs

basics

~10 s

Set a TIMEOUT property, or ctest --timeout as a default, so a hang becomes a reported timeout. Then ctest --repeat until-fail:200 on that test reproduces the rare failure; until-pass and after-timeout only tolerate flakiness.

solid answer

~40 s

Timeouts come from three places: the test's `TIMEOUT` property, which wins; `ctest --timeout <seconds>`, which applies to tests without one; and, when the project uses `include(CTest)`, the `DART_TESTING_TIMEOUT` default of 1500 seconds. A plain `enable_testing()` project has no configured default, so a hang can stall CI until the job itself is killed. A timed-out test is killed and reported as `Timeout`. To **find** a one-in-fifty failure, `ctest --repeat until-fail:200 -R net_reconnect --output-on-failure` runs it until it fails, up to 200 times. `--repeat until-pass:3` and `--repeat after-timeout:2` instead **tolerate** flakiness by retrying, which keeps CI green but hides the bug, so use them knowingly. `--schedule-random` shuffles order to expose hidden dependencies between tests, and since CMake 4.1 `--schedule-random-seed` replays a failing order.

code

cmake · 4 lines
cmake
add_test(NAME net_reconnect COMMAND net_tests reconnect)
set_tests_properties(net_reconnect PROPERTIES
  TIMEOUT 30
  LABELS  "integration;network")

go deeper

for a junior

Know that a TIMEOUT property makes ctest kill and report a hung test instead of waiting.

for a middle

Explain how TIMEOUT, ctest --timeout and the CTest module's default interact, and what each --repeat mode does.

for a senior

Reproduce a rare failure with --repeat until-fail and randomized order, and set timeouts per label so CI never hangs anonymously.

for a principal

Decide whether retries are ever allowed in the merge gate, and require flaky-test reporting so tolerance does not become permanent.

## Hangs first: timeouts A hung test is worse than a failing one: the CI job stalls until the CI system's own limit kills it, and the log rarely says which test was stuck. CTest kills a test that exceeds its timeout and reports it as `Timeout`, so the summary names the culprit. Where the limit comes from, highest precedence first: 1. The **`TIMEOUT` test property**, in seconds: `set_tests_properties(net_reconnect PROPERTIES TIMEOUT 30)`. It takes precedence over everything below. An explicit `0` means no timeout. 2. **`ctest --timeout <seconds>`**, which sets a default for every test that has no `TIMEOUT` property. A test preset's `execution.timeout` is the same setting. 3. With **`include(CTest)`**, the `DART_TESTING_TIMEOUT` cache variable, default `1500`, written into `DartConfiguration.tcl`; ctest applies it even in a plain run. In a CMake 4.4.4 run with it lowered to 2, a 4-second test was reported as `Timeout`. A project that calls only `enable_testing()` writes no `DartConfiguration.tcl`, so the CTest module's default does not apply: `ctest -VV` reported `Test timeout computed to be: 10000000` seconds, effectively none, and the same 4-second test simply passed. For a 300-executable suite, giving each label a sensible `TIMEOUT` (seconds for unit tests, minutes for integration tests) turns a stuck job into a precise report. ## Then the flake: --repeat `--repeat <mode>:<n>` (CMake 3.17+) runs each selected test repeatedly: | Mode | Behaviour | Use | |---|---|---| | `until-fail:<n>` | each test must pass `<n>` times in a row to pass | **finding** sporadic failures | | `until-pass:<n>` | each test may run up to `<n>` times to pass | **tolerating** sporadic failures | | `after-timeout:<n>` | retries only tests that timed out | tolerating timeouts on busy machines | The older `--repeat-until-fail <n>` is equivalent to `--repeat until-fail:<n>`. To chase a one-in-fifty failure locally or on a CI machine: ```console ctest --test-dir build -R '^net_reconnect$' --repeat until-fail:200 --output-on-failure ``` The test stops repeating at its first failure, and `--output-on-failure` prints that attempt's output. Running the repeat alongside the rest of the suite with `-j` adds load, which often makes timing-dependent bugs appear sooner. ## Retrying is a policy decision `until-pass` makes a pipeline green when a test passes on a retry. That is sometimes the right short-term call for a known infrastructure flake, but it has costs: - real intermittent bugs, such as races, ship unnoticed; - the suite gets slower as retries accumulate; - nobody sees the flake rate unless it is reported. Teams that use it usually pair it with a label for known-flaky tests and a report of which tests needed a retry, so the list shrinks instead of growing. ## Order-dependent failures Some tests fail only after another particular test, because the first leaves a file or setting behind. `--schedule-random` runs tests in a random order to expose such hidden dependencies. Since CMake 4.1, `--schedule-random-seed <seed>` reruns the same order, which turns a lucky failure into a reproducible one. Once found, the fix is to remove the shared state, or to declare it with fixtures or `RESOURCE_LOCK`. ## Reading a timeout correctly A `Timeout` result does not prove the code deadlocked. Common causes, roughly in order of likelihood: 1. The limit is too tight for a loaded CI machine, especially under a high `-j`. 2. The test waits on something external, such as a network peer or a lock held by another test. 3. The code under test really is stuck. `ctest -VV` prints the computed timeout for each test, which settles which of the three precedence levels applied. Running the single test with `--repeat until-fail` and no `-j` separates cause 1 from causes 2 and 3. ## Putting it into CI - Give every test a `TIMEOUT`, or run CI with `--timeout`, so no job can hang silently. - Run with `--output-on-failure` and `--output-junit` so timeouts and failures carry their output. - Keep a scheduled job that runs suspect labels with `--repeat until-fail:<n>` and `--schedule-random`, separate from the merge gate. Why a given test races, and how to fix the code, belongs to the material on concurrency and on the test framework in use.

  • Why might a test that never sets TIMEOUT still be killed after 25 minutes?
    If the project uses `include(CTest)`, the CTest module defines `DART_TESTING_TIMEOUT` with a default of 1500 seconds and writes it into `DartConfiguration.tcl`; ctest applies it to tests without their own `TIMEOUT` property. Raise or lower it with `-DDART_TESTING_TIMEOUT=...` at configure time, or give the test an explicit `TIMEOUT`.
  • What is the difference between --repeat until-fail:50 and --repeat until-pass:50?
    `until-fail:50` requires each test to pass 50 runs in a row, stopping at the first failure, so it surfaces rare failures. `until-pass:50` lets each test try up to 50 times and counts it as passing once any attempt passes, so it hides rare failures. The first is a diagnostic tool; the second is a tolerance policy.

  • ctest --timeout overrides a test's own TIMEOUT property
  • Every CTest project gets a 1500-second default timeout
  • --repeat until-pass is a good way to find flaky tests
  • A hung test makes ctest wait forever regardless of settings
  • --schedule-random alone lets you reproduce a failing order exactly