When would you choose runtime-generated JUnit 5 tests from a @TestFactory over statically declared test methods in a large suite, and what does that choice cost you in tooling and CI?

## The decision rule Ask one question: *could I have written these test methods by hand at authoring time and been correct?* - **Yes** — write them, or use a statically declared data-driven mechanism. You keep the full tooling contract. - **No, because the cases come from the environment** — a factory is the right tool, and the maintenance win is real. The strongest cases for dynamic generation share one property: **the list would otherwise be hand-maintained and would silently rot**. - One contract test per implementation discovered through a service registry — add an implementation, it is tested automatically; forget to register it and the registry test catches that separately. - One case per fixture file in a directory — a new fixture becomes a case with no code change, which is exactly what you want from a corpus of regression samples. - One case per entry in a specification the build fetches, so the suite tracks the spec rather than a copy of it. - A matrix over two runtime-derived collections, where the cross product cannot be enumerated in source. ## What you trade away **Pre-run visibility.** Discovery builds the test plan before execution; dynamic cases are created during execution. Practical effects: the IDE shows a single factory node until you run it; you cannot click one generated case to run it in isolation; pre-run filters and unique-ID selection cannot target a case that does not yet exist. Debugging a single failing case means running the whole factory, often with a temporary filter inside the factory itself. **Lifecycle and extensions.** `@BeforeEach`/`@AfterEach` and their extension equivalents wrap the factory method, not each case. All cases share one instance and one fixture, so isolation is hand-rolled inside each case body. Teams that adopt factories broadly reinvent a small per-case setup helper — a sign the mechanism is being pushed past its intent. **Identity stability.** Dynamic cases are identified positionally within the factory. Insert a case in the middle and every later case's identifier shifts. Anything keyed on identifiers — rerun-failed lists, flaky-test dashboards, quarantine lists, historical duration tracking — mis-attributes results. On a suite with flaky-test automation this is a genuine operational cost. **Volume control.** A factory over a large corpus can emit thousands of cases. Lazy consumption keeps memory fine, but reports, dashboards and log scrapers may not cope, and the signal-to-noise of a run degrades. You end up designing grouping (containers), sampling, or a nightly-full/per-commit-subset split — real work that static suites get for free from their fixed size. **Silent zero.** The failure mode that actually bites teams: empty input produces zero tests and a passing build. A fixture directory dropped from the build output, a filter that matches nothing, a registry that fails to load — each quietly deletes a suite. Any factory reading external input needs an explicit non-empty assertion, and ideally a CI check on total test count. **Cognitive load.** A reader of a static test knows exactly what runs. A reader of a factory must mentally execute the generation logic to know what the suite covers, and the answer can differ per environment. Keep generation logic trivial — read, map, name — and push complexity into the assertions. ## Guardrails I would set for a team 1. Factories are for **runtime-derived case sets**; looping over a checked-in list uses static declarations. 2. Every factory reading external input **asserts non-empty** before generating. 3. Display names must **identify the case** (file, ID, key) — never a bare index. 4. Generation logic stays **trivial and side-effect free**; per-case fixtures live inside the case body. 5. CI fails on a **drop in total executed tests**, which catches both empty factories and silently undiscovered classes. 6. Large corpora are **grouped into containers** and, if necessary, split into a fast subset and a full nightly run. ## How to answer this well A principal-level answer does not say "dynamic tests are powerful". It states the narrow condition that justifies them, names the specific tooling contracts that break (discovery-time plan, per-case lifecycle, stable identity), and closes with the operational guardrails that make the tradeoff safe — because the risk here is not a wrong test, it is a suite that quietly stops testing anything.