skip to content

What are the common pitfalls of test selection — pattern semantics, the JUnit 4 vs Platform divide, and silently dropped tests?

level: seniorimportance: should knowfreq 26%

answer

  1. glob *, not regex
  2. tags inert under useJUnit()
  3. rename = silent un-exclude
  4. failOnNoMatchingTests default true
  5. class-level @Tag covers all methods

basics

~20 s

Pitfalls: assuming patterns are regex (they're glob * only); using @Tag/includeTags while still on useJUnit() (JUnit 4 ignores them); and name-based excludes silently missing tests after renames. Also failOnNoMatchingTests defaulting on can break filtered runs.

solid answer

~40 s

Key traps: (1) **Pattern semantics** — `includeTestsMatching` uses glob `*`, not regex; `OrderTest.*` won't behave like a regex. (2) **Engine mismatch** — `@Tag` and `includeTags/excludeTags` only work under `useJUnitPlatform()`; on a JUnit 4 task (`useJUnit()`) they are inert, and you'd use `@Category` + `includeCategories`. (3) **Silent drops** — a name-based `excludeTestsMatching("*SlowTest")` stops excluding a renamed class without error, regressing CI time or coverage. (4) **failOnNoMatchingTests** is `true` by default, so an over-narrow filter fails the build; set it false where empty is legitimate. (5) Tag filters apply at discovery, so a class with `@Tag` at class level applies to all its methods unless overridden. Mitigations: prefer tags for stable categories, keep `failOnNoMatchingTests` on to catch typos, and confirm the active engine before relying on tag filtering.

code

kotlin · 8 lines
kotlin
// Wrong: regex-style pattern, won't match as intended
filter { includeTestsMatching("Order(Test|Spec)") }

// Right: glob with *
filter { includeTestsMatching("*OrderTest") }

// JUnit 4 task: tags are ignored; use categories instead
tasks.test { useJUnit { includeCategories("com.acme.SlowTests") } }

go deeper

for a junior

Know patterns are glob not regex, and tags need JUnit Platform.

for a middle

Explain the silent-drop rename problem and the failOnNoMatchingTests default.

for a senior

Diagnose engine mismatch in a multi-module build and choose tags vs name filters to avoid silent regressions.

for a principal

Mandate tag-based categorization and JUnit Platform migration to eliminate the JUnit 4 inert-tag class of bugs org-wide.

## Pitfall 1 — patterns are not regular expressions `includeTestsMatching`/`excludeTestsMatching` accept **glob-like** patterns whose only wildcard is `*`. People paste regex like `Order(Test|Spec)` or `.*Test` expecting it to work; `.*Test` matches a literal dot, and the alternation does nothing. Use `*Test`, `*.OrderTest`, etc. ## Pitfall 2 — the JUnit 4 vs Platform divide Tag filtering (`includeTags`/`excludeTags`, `@Tag`) is a **JUnit Platform** feature. If the `Test` task still calls `useJUnit()` (classic JUnit 4) the tag config is silently ignored — no error, tests just don't filter. The JUnit 4 equivalent is marker-interface categories: ```kotlin tasks.test { useJUnit { includeCategories("com.acme.SlowTests") } } ``` Always confirm the engine (`useJUnitPlatform()`) before relying on tags. In a multi-module build, one module on JUnit 4 can quietly run tests you thought were excluded. ## Pitfall 3 — silent drops from name coupling Name filters couple to conventions. Rename `PaymentSlowTest` → `PaymentHeavyScenario` and `excludeTestsMatching("*SlowTest")` stops excluding it. Because other `*SlowTest` classes still exist, `failOnNoMatchingTests` doesn't fire, so the slow test now runs in your fast loop with no warning. Tags (`@Tag("slow")`) avoid this — the label moves with the method. ## Pitfall 4 — failOnNoMatchingTests ```kotlin tasks.test { filter { includeTestsMatching("*NonexistentTest") // isFailOnNoMatchingTests = true (default) -> build FAILS } } ``` The default `true` is usually a feature (catches typos) but bites when a filter legitimately matches nothing in some modules. Set it `false` there rather than globally. ## Pitfall 5 — class-level tag inheritance A `@Tag("slow")` on the class applies to every method. If you only meant one method slow, tag the method, or you'll exclude the whole class under `excludeTags("slow")`. ## Pitfall 6 — combining filters surprises Name filter (Gradle layer) and tag filter (engine layer) **intersect**. Adding a name include on a tag-filtered task can shrink the set unexpectedly to the AND of both. ## Mitigations checklist - Prefer `@Tag` over `*Name*` for categories that drive CI. - Keep `failOnNoMatchingTests = true` to surface typos; override only where empty is valid. - Verify `useJUnitPlatform()` is active before relying on tags. - Tag at method level when only some methods qualify.

  • Your `excludeTags("slow")` isn't excluding anything — what do you check first?
    Whether the task uses `useJUnitPlatform()`. Under `useJUnit()` (JUnit 4) tags are ignored; you'd need `@Category` + `includeCategories`.
  • How can a name-based exclude silently stop working?
    A class rename can make the pattern no longer match; since other classes still match, `failOnNoMatchingTests` doesn't fire and the test silently runs.
  • When would you disable failOnNoMatchingTests?
    When a shared filter legitimately matches zero tests in some modules; set it false only there, keeping the typo-catching default elsewhere.

saying these in an interview costs you the question

  • Writing regex in `includeTestsMatching` and expecting it to work.
  • Relying on `@Tag` while the task is still on `useJUnit()`.
  • Blindly setting `failOnNoMatchingTests = false` globally to dodge a real typo.

context

resolving links…