skip to content

Your CI shows green even when tests fail to compile or the build crashes before tests run. How should CI consume Gradle's JUnit-XML so test results are surfaced correctly?

level: seniorimportance: must knowfreq 48%

answer

  1. exit code = gate, XML = display
  2. if: always() / post always
  3. **/build/test-results/**/*.xml glob
  4. no XML on compile failure = false green
  5. mergeReruns for flaky

basics

~10 s

Point the CI test-reporter at **/build/test-results/test/*.xml, upload it as an artifact, and make sure the publish step runs even when the Gradle step fails (e.g. if: always()) so failures and crashes are still reflected.

solid answer

~50 s

CI tools (GitHub Actions test-reporters, Jenkins JUnit plugin, GitLab `junit:` artifacts) parse Gradle's JUnit-XML at `build/test-results/<task>/TEST-*.xml`. Three rules matter: 1. **Glob across modules** — `**/build/test-results/**/*.xml` so multi-project builds are covered. 2. **Always run the publish step**, even on failure (`if: always()` in GH Actions, `post { always { junit ... } }` in Jenkins). Otherwise a failing Gradle step short-circuits and CI never reads the XML, masking the real result. 3. **Don't let an empty/missing report read as success.** If compilation fails, no XML is produced at all — your reporter must treat 'no results' as a problem, and the Gradle step's non-zero exit must still fail the job. Rely on Gradle's exit code as the source of truth and use the XML only for *display*, not for the pass/fail decision. Enabling `junitXml.mergeReruns` lets the reporter mark flaky retried tests instead of double-counting them.

code

yaml · 9 lines
yaml
- name: Run tests
  run: ./gradlew test --continue
- name: Publish test report
  if: always()
  uses: dorny/test-reporter@v1
  with:
    name: JUnit Tests
    path: '**/build/test-results/test/TEST-*.xml'
    reporter: java-junit

go deeper

for a junior

Know CI reads the TEST-*.xml files to show results.

for a middle

Point the reporter at the right glob and upload it as an artifact.

for a senior

Separate exit-code gating from XML display, ensure always-publish, handle multi-module + flaky.

for a principal

Define the org-wide CI contract: exit-code is truth, XML is annotation, mandatory always-publish to prevent false-green across all pipelines.

## The failure mode The classic 'green when it should be red' bug happens because teams treat the **JUnit-XML report** as the source of truth for pass/fail. But XML is only written for tests that actually *ran*. If `compileTestJava`/`compileTestKotlin` fails, or the JVM crashes, or a plugin errors before the `test` task — **no XML exists**, or stale XML from a previous run remains. A naive reporter that 'parses whatever XML it finds and passes if no failures are present' will report success. ## Two independent signals Keep these separate: - **Pass/fail decision = the Gradle process exit code.** `./gradlew test` exits non-zero on any failure, compile error, or crash. CI must let that exit code fail the job. Never wrap it in something that swallows the code. - **Display/annotations = the JUnit-XML.** Reporters read `build/test-results/<task>/TEST-*.xml` to render per-test annotations, history, and flaky markers. This is presentation, not gating. ## Making the publish step robust The reporting step must run **even when the build step failed**, otherwise a red build produces no annotations: ```yaml - name: Run tests run: ./gradlew test - name: Publish test report if: always() # <- crucial uses: dorny/test-reporter@v1 with: name: JUnit Tests path: '**/build/test-results/test/TEST-*.xml' reporter: java-junit ``` Jenkins equivalent: ```groovy post { always { junit '**/build/test-results/**/*.xml' } } ``` GitLab: ```yaml artifacts: when: always reports: junit: '**/build/test-results/test/TEST-*.xml' ``` ## Multi-module globbing In a multi-project build each subproject writes its own `:<sub>:test` output under `<sub>/build/test-results/test/`. Use a recursive glob (`**/build/test-results/...`) so every module's XML is collected. ## Stale results Gradle is incremental: if `test` is `UP-TO-DATE` it won't rewrite XML, and a `--rerun`/clean may be wanted when you must regenerate. For caching/up-to-date runs the XML simply persists from the last execution — fine, because the exit code is still 0. The danger is only when a *new* failure occurs upstream of the task. ## Flaky handling With `junitXml.mergeReruns = true` plus a retry mechanism (e.g. the test-retry plugin), retried executions collapse into `<flakyFailure>` entries. Reporters that understand this won't count a test that eventually passed as a hard failure, and can surface it as flaky instead. ## Summary rule **Gate on the exit code; annotate from the XML; always publish.** That ordering eliminates the false-green class of bugs.

  • Why must the report-publishing step use `if: always()`?
    A failing `./gradlew test` step would otherwise short-circuit the job and skip publishing, leaving the failure unannotated. `if: always()` ensures the XML is still parsed and surfaced even on red builds.
  • Why shouldn't CI decide pass/fail purely from the XML?
    Compile errors or crashes produce no XML (or stale XML). A reporter that 'passes when it finds no failing testcase' goes green falsely. The authoritative signal is Gradle's non-zero exit code.
  • What does `--continue` change about the results?
    It lets Gradle run all test tasks even after one fails, so you get complete XML across modules in one run; the overall build still exits non-zero.

saying these in an interview costs you the question

  • Treating presence/absence of failing testcases in XML as the build's pass/fail gate
  • Publishing only on success so red builds show no test annotations
  • Globbing a single module's results in a multi-project build

context

resolving links…