skip to content

When is `--tests` the right tool versus committed filter configuration or test-suite splitting, and how does that shape a team's local-vs-CI test strategy?

level: seniorimportance: should knowfreq 25%

answer

  1. CLI = ephemeral iteration
  2. build script = durable policy
  3. suites/source sets for unit vs IT
  4. CI sharding != --tests
  5. don't let CI depend on typed --tests

basics

~20 s

--tests is a transient, local-iteration tool you type by hand to run a focused subset. Durable selection (which suites run, tags, splits) belongs in the build script or CI config, not in an ad-hoc CLI flag.

solid answer

~40 s

`--tests` exists for fast, throwaway local feedback: a developer narrows execution to one class/method while writing code, without touching the build. It is **not committed** and not a strategy. Anything that must persist — running only `@Tag("fast")` tests, separating unit from integration suites, sharding across CI agents — belongs in code: `test { useJUnitPlatform { includeTags("fast") } }`, dedicated `Test` tasks/source sets or the JVM Test Suite plugin, and CI parallelism (matrix/shards). The mental model: CLI `--tests` = ephemeral developer iteration; build-script filters/suites = reproducible, reviewable policy that every machine and CI run honors. Mixing them up (e.g. relying on developers to type the right `--tests` in CI, or hardcoding includes that hide tests) leads to non-reproducible or silently-skipped coverage.

code

kotlin · 12 lines
kotlin
// durable, committed selection — not a CLI flag
tasks.test {
    useJUnitPlatform {
        includeTags("fast")
        excludeTags("slow")
    }
}

// a separate lane for slower integration tests
val integrationTest by tasks.registering(Test::class) {
    useJUnitPlatform { includeTags("slow") }
}

go deeper

for a junior

Know that --tests is for local one-off runs and not committed.

for a middle

Contrast ephemeral CLI selection with build-script filters/tags and separate test tasks.

for a senior

Lay out the layers (CLI vs filters vs suites vs CI sharding) and the anti-patterns when they're conflated.

for a principal

Define org policy: where selection lives, how lanes/suites are structured, and how reproducibility and review are guaranteed.

## Two layers of selection Gradle offers selection at different durabilities, and choosing the right layer is a design decision: ### 1. Ephemeral — `--tests` (CLI) Typed per invocation, never committed. Perfect for **local iteration**: you're editing `OrderService` and want only `OrderServiceTest.placesOrder` to run on each save. It applies on top of whatever the build already configures. ### 2. Durable — build-script filters & tags Committed, reviewed, reproducible. Configured on the `Test` task: ```kotlin tasks.test { useJUnitPlatform { includeTags("fast") excludeTags("slow") } filter { includeTestsMatching("com.example.*Test") isFailOnNoMatchingTests = false } } ``` Every developer and every CI run gets the same behavior. ### 3. Structural — suites & source sets For unit vs. integration vs. functional, create separate `Test` tasks (or use the **JVM Test Suite plugin**, `testing { suites { ... } }`). This lets CI run `test` on every push and `integrationTest` on a slower lane. ### 4. Distribution — CI sharding/parallelism Splitting a large suite across agents is a CI concern (build matrix, test-distribution tooling), layered on top of suites — not something `--tests` should encode. ## Choosing the layer | Need | Right tool | |------|-----------| | Run one class while coding | `--tests` (CLI) | | Always run only fast-tagged tests | `includeTags` in build | | Separate unit & integration lanes | suites / source sets | | Spread suite across CI agents | CI sharding | ## Anti-patterns - **CI depending on hand-typed `--tests`** — non-reproducible; whatever a script author typed becomes the de-facto policy. - **Hardcoded `includeTestsMatching` that hides tests** — coverage silently drops; new tests outside the pattern never run. Pair with `isFailOnNoMatchingTests` awareness and code review. - **Using `--tests` to permanently skip flaky tests** — that belongs in tags/quarantine, visible in source. ## Why it matters The principle is reproducibility and reviewability: durable policy lives in version control where it's diffable; `--tests` stays a personal, momentary convenience. Keeping the layers distinct prevents the classic "works on my machine / passed in CI but skipped half the suite" failures.

  • Why is it a smell for a CI pipeline to invoke `./gradlew test --tests '...'`?
    Selection then lives in an ad-hoc command rather than reviewable build config, so it's easy to silently exclude tests, hard to reproduce locally, and changes bypass code review. Durable selection should be in the build script or suite definitions.
  • How would you keep slow integration tests out of the fast feedback loop without `--tests`?
    Tag them (`@Tag("slow")`) and either exclude that tag from the `test` task or put them in a separate `integrationTest` task/suite that runs on its own CI lane.

saying these in an interview costs you the question

  • Treating `--tests` as a committed test-selection strategy.
  • Encoding which tests CI runs by hardcoding `--tests` patterns in pipeline scripts.
  • Using `--tests` to permanently hide flaky tests instead of tagging/quarantining them.

context

resolving links…