Tests in a CMake suite pass with plain ctest but fail under ctest -j 8 because several share one scratch database; which test properties make them safe to parallelize?
answer
- serialize vs set up
- RESOURCE_LOCK names a shared thing
- FIXTURES_SETUP, REQUIRED, CLEANUP
- DEPENDS is order only
- RUN_SERIAL as last resort
basics
~20 sGive the database tests one RESOURCE_LOCK so ctest never runs them together, and use FIXTURES_SETUP, FIXTURES_REQUIRED and FIXTURES_CLEANUP so creating and deleting it run first and last. DEPENDS only orders tests; RUN_SERIAL is the blunt fallback.
solid answer
~40 sParallel failures that vanish serially mean tests share state. CTest offers properties for each need. **`RESOURCE_LOCK db`** on every test that uses the database guarantees no two of them run at once, while unrelated tests still run in parallel. **Fixtures** handle setup and teardown: `db_start` gets `FIXTURES_SETUP db`, `db_stop` gets `FIXTURES_CLEANUP db`, and each query test gets `FIXTURES_REQUIRED db`; ctest runs setup once before them and cleanup after them, skips them if setup fails, and adds setup and cleanup automatically when you select a subset with `-R`. `DEPENDS` only orders tests, ignoring whether the earlier ones passed, and `RUN_SERIAL` stops a test running alongside anything, which wastes the other cores. The better fix is often to remove the sharing: give each test its own `WORKING_DIRECTORY` or a unique temporary path.
code
cmake · 10 linesadd_test(NAME db_start COMMAND ${CMAKE_COMMAND} -E make_directory db)
add_test(NAME db_stop COMMAND ${CMAKE_COMMAND} -E rm -rf db)
add_test(NAME query_users COMMAND ${CMAKE_COMMAND} -E chdir db ${CMAKE_COMMAND} -E true)
add_test(NAME query_orders COMMAND ${CMAKE_COMMAND} -E chdir db ${CMAKE_COMMAND} -E true)
set_tests_properties(db_start PROPERTIES FIXTURES_SETUP db)
set_tests_properties(db_stop PROPERTIES FIXTURES_CLEANUP db)
set_tests_properties(query_users query_orders PROPERTIES
FIXTURES_REQUIRED db
RESOURCE_LOCK db)go deeper
Know that ctest -j runs tests at the same time, so tests that share files or ports can collide.
Explain RESOURCE_LOCK, the three fixture properties, and why DEPENDS is only an ordering.
Find the shared resource behind a parallel-only failure, remove it where possible, and otherwise lock or fixture it without serializing the whole suite.
Set a team rule that tests own their scratch paths and ports, so the suite scales with -j instead of accumulating locks.
## Diagnosing a parallel-only failure When a suite passes with plain `ctest` (one test at a time) and fails with `ctest -j 8`, the tests are not independent. Typical shared state: - one scratch database or directory, such as `db/` in the build tree; - a fixed TCP port; - one GPU or hardware device; - a file that one test writes and another reads. CTest runs each test as a separate process, so in-memory state is never shared; the problem is always something outside the process. Before reaching for properties, ask whether the sharing is necessary. Giving each test its own `WORKING_DIRECTORY`, or a path built from the test name, removes the conflict entirely and keeps full parallelism. ## Serializing access: RESOURCE_LOCK `RESOURCE_LOCK` takes a list of resource names. Tests that name the same resource are **guaranteed not to run concurrently**; everything else still runs in parallel. ```cmake set_tests_properties(render_a render_b PROPERTIES RESOURCE_LOCK gpu) ``` In a CMake 4.4.4 run of two one-second tests with that lock, `ctest -j8 -R render` took about 2.1 seconds of real time: one after the other. The name `gpu` is arbitrary; it only has to match between tests. ## Setup and teardown: fixtures Fixtures (CMake 3.7+) model "something must exist before these tests and be removed after them": | Property | Set on | Meaning | |---|---|---| | `FIXTURES_SETUP db` | the test that creates the resource | runs before any test requiring `db` | | `FIXTURES_REQUIRED db` | each test that needs it | runs only after setup succeeded | | `FIXTURES_CLEANUP db` | the test that removes it | runs after every test requiring `db` has finished | The documented guarantees: 1. Setup tests run **once** for the whole set of requiring tests, not once per test. 2. If any setup test fails, the tests requiring the fixture are **not run**. 3. Cleanup tests **always** run, even after a failed setup. 4. When ctest runs a subset (`-R`, `--rerun-failed`), it **adds** the required setup and cleanup tests automatically. A CMake 4.4.4 run of `ctest -R query_users` executed `db_start`, `query_users` and `db_stop`. The options `-FA`, `-FS` and `-FC` switch that off. Fixture names and lock names are independent. A database that must be created once and accessed by one test at a time needs both: `FIXTURES_REQUIRED db` and `RESOURCE_LOCK db` on each query test. ## Weaker or blunter tools - **`DEPENDS`** sets a run-after order only. The documentation is explicit that the earlier tests' results are ignored, so a query test still runs after a failed setup. Use fixtures when success matters. - **`RUN_SERIAL`** makes a test run with no other test at the same time. It is correct but costly on a 300-test suite: every core but one idles while it runs. - **`PROCESSORS`** tells ctest how many of the `-j` slots a multi-threaded test uses, so `-j 8` does not start eight tests that each spawn eight threads. ## Choosing between the tools | Need | Property | Effect on parallelism | |---|---|---| | tests must not overlap on one resource | `RESOURCE_LOCK` | only the lock holders serialize | | a resource must exist first and be removed after | `FIXTURES_SETUP`, `FIXTURES_REQUIRED`, `FIXTURES_CLEANUP` | none beyond the ordering | | a test must run after another, whatever its result | `DEPENDS` | ordering only | | a test must have the machine to itself | `RUN_SERIAL` | blocks everything while it runs | | a test uses several cores | `PROCESSORS` | takes that many `-j` slots | A useful habit is to start from the cheapest option that is correct: unique paths first, then a lock, then fixtures, and `RUN_SERIAL` only for tests that truly need the whole machine. ## Timeouts and labels in the same pass While tests are being classified, two more properties belong with them: - **`TIMEOUT`** in seconds kills a hung test and reports it as a timeout instead of stalling the pipeline; it takes precedence over `ctest --timeout` and the CTest module's default. - **`LABELS`** such as `integration` or `db` let CI run groups separately, for example `ctest -L db -j 2` alongside `ctest -LE db -j 16`. Test properties can only be set in the directory that created the test, so a helper function that calls `add_test` should set them in the same place.
- When is RUN_SERIAL the right choice instead of RESOURCE_LOCK?When a test disturbs everything rather than one named resource, such as a benchmark that needs the whole machine quiet, or a test that changes global system settings. `RESOURCE_LOCK` only excludes tests naming the same resource, so it cannot protect against unknown neighbours; `RUN_SERIAL` can, at the price of idling other cores while it runs.
- A setup test for a fixture fails; what happens to the tests that require it and to its cleanup test?The tests listing the fixture in `FIXTURES_REQUIRED` are not executed, so they are reported as not run rather than as misleading failures. The fixture's cleanup tests still run, so a half-created resource is removed. Plain `DEPENDS` would have run the dependent tests anyway, because it ignores results.
- DEPENDS makes a test skip when the test it depends on fails
- RESOURCE_LOCK runs the fixture's setup test automatically
- Fixture setup tests run once before every requiring test
- Tests in one ctest run share memory, so globals leak between them
- RUN_SERIAL is the standard fix for any parallel flakiness