skip to content

A Catch2 test calls GENERATE(1, 2), then has SECTION one with GENERATE(-3, -2) and SECTION two with GENERATE(4, 5, 6). How many passes run, and why?

level: seniorimportance: nice to knowfreq 15%

answer

  1. a generator is an implicit section
  2. it reaches to the end of its scope
  3. multiply inside each branch
  4. then add the branches

basics

~20 s

Ten passes. GENERATE acts as an implicit section reaching to the end of its scope, so each outer value replays every leaf below it: section one runs 2 x 2 = 4 times, section two 2 x 3 = 6.

solid answer

~40 s

Catch2's documentation describes `GENERATE` as an implicit `SECTION` that runs from where it is written to the end of its scope. The outer `GENERATE(1, 2)` therefore wraps both sections, and each of its values replays the whole tree below it. Inside section one, `GENERATE(-3, -2)` needs two passes per outer value, so one runs four times; inside section two, `GENERATE(4, 5, 6)` needs three, so two runs six times: ten passes, with the outer value fixed while both sections are exhausted. Placement matters: a `GENERATE` between two sections replays only the code after it, so a first section runs once while a second runs once per value. These flows grow fast, and filtering a single element with `-c` or the generator filters added in 3.13.0 still exhausts generators that are not filtered.

code

cpp · 23 lines
cpp
#include <catch2/catch_test_macros.hpp>
#include <catch2/generators/catch_generators.hpp>
#include <iostream>

TEST_CASE("generators inside sections", "[gen]") {
    int i = GENERATE(1, 2);
    SECTION("one") {
        int j = GENERATE(-3, -2);
        std::cout << "one i=" << i << " j=" << j << '\n';
        REQUIRE(j < i);
    }
    SECTION("two") {
        int k = GENERATE(4, 5, 6);
        std::cout << "two i=" << i << " k=" << k << '\n';
        REQUIRE(i != k);
    }
}

TEST_CASE("generate between sections", "[gen]") {
    SECTION("first") { std::cout << "first\n"; REQUIRE(true); }
    auto value = GENERATE(1, 2);
    SECTION("second") { std::cout << "second " << value << '\n'; REQUIRE(true); }
}

go deeper

for a junior

Recall that GENERATE and SECTION both cause the test case to be re-entered, and that the number of passes can grow quickly.

for a middle

Count passes for a case with one generator and a few sections, and explain why sibling sections add while nested generators multiply.

for a senior

Read GENERATE as an implicit section to the end of its scope, place generators to avoid replaying unrelated work, and narrow a failing combination with section and generator filters.

for a principal

Set limits on generator products in shared suites, because pass counts that explode quietly turn into CI minutes and slow feedback for everyone.

## One model for sections and generators Catch2 runs a test case as a series of **passes**, re-entering it from the top until every path has been taken. Sections and generators are both nodes on those paths. The generator documentation puts it directly: `GENERATE` can be seen as an **implicit `SECTION`** that goes from the place it is used to the end of the scope. Once that is understood, counting passes is arithmetic. ## Counting the example ```cpp TEST_CASE("generators inside sections", "[gen]") { int i = GENERATE(1, 2); SECTION("one") { int j = GENERATE(-3, -2); REQUIRE(j < i); } SECTION("two") { int k = GENERATE(4, 5, 6); REQUIRE(i != k); } } ``` 1. The outer generator spans both sections, so everything below it is replayed for `i == 1` and again for `i == 2`. 2. For one value of `i`, section one contains a generator of two values: two passes. 3. For one value of `i`, section two contains a generator of three values: three passes. 4. Per outer value that is 2 + 3 = 5 passes; for two outer values, **10 passes** and 10 assertions. A run printed them in this order: ```plaintext one i=1 j=-3 one i=1 j=-2 two i=1 k=4 two i=1 k=5 two i=1 k=6 one i=2 j=-3 ... ``` The outer value stays fixed while every branch beneath it is exhausted, then advances, which matches the documented order of one, one, two, two, two, one and so on. ## Placement changes the count Because the implicit section starts at the `GENERATE`, code above it is not replayed per value: ```cpp TEST_CASE("generate between sections", "[gen]") { SECTION("first") { REQUIRE(true); } auto value = GENERATE(1, 2); SECTION("second") { REQUIRE(true); } } ``` This reports **3 assertions**: `first` runs once and `second` runs once per value. Placing a generator between sections has been allowed since Catch2 2.13.0. The documentation's own example, with generators of two values before, between and after two sections, reports 14 assertions, which shows how quickly such flows stop being obvious. ## Rules for counting - A generator multiplies **everything after it in its scope**, including nested sections and later generators. - Sibling sections **add**: each is a separate branch, and each branch carries its own product. - A generator inside a section multiplies **only that section's** subtree. - Code before a generator runs once per pass of whatever encloses it, not once per generator value. ## Filtering one path The runner can narrow a run to one path, which helps when a single combination fails: - `-c "two"` selects a section by name; nested sections take repeated `-c` options. - Catch2 3.13.0 added `-g` (`--generator-index`) to pick one generator element and `-p` (`--path-filter`) for generic path filters; all of them form one stack of filters. - A generator that is **not** filtered is still exhausted. The documentation's examples show that selecting a section after an unfiltered generator still runs once per generator element. ## Designing with both - Put a generator **inside** the section that needs it, so other sections are not replayed for values they ignore. - Put shared, value-independent set-up **above** the first generator. - Keep at most one level of generator nesting in a test case; deeper products belong in separate cases with clear names. - Count passes before review rather than after a slow CI run; a short comment with the expected number makes the intent checkable. ## Common surprises - **Expensive set-up in a parent section** above a generator runs once per generated value, not once overall. - **A generator inside a loop** is a single generator at one source location, so it does not create a new generator per iteration; generators belong at a fixed point of the case, not inside loops. - **Random generators** multiply in the same way, and a different seed changes the values but not the number of passes. - **Assertion counts** in the summary grow with every pass, which is a quick way to confirm the arithmetic: the example above reported 13 assertions for its two test cases. ## Why this matters in practice A team migrating a suite to Catch2 3 often introduces generators to replace hand-written loops. The loop version ran each block once; the generator version may replay unrelated sections per value, multiplying run time and, worse, re-running expensive set-up that sat above the generator in a parent section. Reading the generator as a section boundary is what prevents both surprises.

  • Why does code above a Catch2 GENERATE not run once per generated value?
    Because the generator's implicit section starts where `GENERATE` is written. Catch2 replays the path from the top on every pass, but the passes that exist to exhaust the generator are counted only for what follows it; a section that ends before the generator completes once and is skipped on later passes, so it does not repeat per value.
  • In Catch2, how many passes does a test case need with GENERATE(1, 2, 3) at the top and three sibling sections with no generators?
    Nine. The generator spans all three sections, and each value of the generator replays every leaf beneath it. Three leaf sections times three values gives nine passes, and the code above the generator still runs on every one of them.

  • A GENERATE at the top of a test case runs once, before the sections.
  • Generators inside sibling sections multiply with each other.
  • Moving a GENERATE between two sections changes nothing.
  • Selecting one section with -c skips every generator in the case.