Maven Surefire writes one JUnit-XML file per test class, named `TEST-<sourceName>.xml`, and each file's `<testsuite>` carries its own `@tests`, `@failures`, `@errors` and `@skipped`. Where does a run-level total come from, and what happens to it when one of those files is never written?
answer
- one file per class, counters per file
- the run total is arithmetic
- nothing declares how many files
- a missing file is not a failure
- zero files sums to green
basics
~20 sNo file holds a run total. Each TEST-<sourceName>.xml declares counters for its own suite only, so the run figure is a consumer's sum over the glob - and a file that was never written drops out of that sum silently.
solid answer
~40 sThe output is one file per class, matched by a `TEST-*.xml` glob, and each file's `<testsuite>` counters describe that file alone. A run-level total exists only as arithmetic the consumer performs over whatever the glob matched. Nothing in the file set says how many files there should be, so a class whose file never arrived - the process died, the upload dropped a path, the glob ran early - simply does not appear. Its cases are absent rather than failed, absence has no representation in this format, and the resulting smaller total is internally consistent and still green. The degenerate version is worse: an empty glob sums to zero tests and zero failures, which most consumers render as a clean run. The expectation you compare against has to come from outside the files.
go deeper
Know that these result files come one per class under a TEST-*.xml glob, and that each file's counters describe only its own suite.
Explain that a run-level total is computed by the consumer over the glob, since neither a file nor the directory declares how many files should exist.
Diagnose the real incident: a class whose file never landed, a smaller but perfectly consistent green total, and the out-of-band checks that would have caught it.
Decide what evidence a release gate accepts, given that a consolidated report can only describe the files that arrived and never the run that was supposed to happen.
## One file per class, counters per file Maven Surefire writes its JUnit-XML output as one file per test class, named `TEST-<sourceName>.xml`. That is a glob, not a fixed name: a consumer finds the results by matching `TEST-*.xml` in the reports directory, and a run of two hundred classes leaves two hundred files. Each file has one `<testsuite>` element carrying its own required counters -- `@tests`, `@errors`, `@skipped` and `@failures` -- which describe that file's suite and nothing else. ## The total that nobody writes down There is no run-level figure anywhere in that set of files. Not in any single file, and not in the directory. The tests-run and failures line on a dashboard is arithmetic performed by the consumer: glob the directory, parse each file's suite header, add the columns up. Every tool doing this computes its own answer from the same inputs, which is fine as long as the inputs are complete. The inputs being complete is precisely what the format cannot tell you. ## Why a missing file is invisible Nothing in the file set declares how many files there should be. Each file is self-describing and internally consistent; the collection is described by nothing at all. So when one file is not written -- the process running that class died, the disk filled, the artefact upload dropped a path, the glob ran before the writer flushed -- the sum is not wrong in any detectable way. It is a smaller, perfectly coherent total in which that class simply never existed. That is the failure mode worth internalising: - **A class that never reported does not fail the run; it vanishes from it.** Its cases are absent, not failed, and absence has no representation in this format. - **The total still adds up.** `@tests` across the surviving files agrees with the surviving `<testcase>` elements, so every internal consistency check you run passes. - **The trend line barely moves.** A total that drops by a few dozen out of thousands looks like somebody deleted a test class, and a run that was green stays green. ## The empty-glob trap The degenerate case of the same problem is worse. Point the consolidation step at the wrong directory, or run it before the tests, and the glob matches nothing. Summing an empty set gives zero tests, zero failures and zero errors -- which most consumers render as a clean run rather than as an error. A gate that asks "were there any failures?" answers no and lets the build through. A gate that asks "did we see the tests we expected?" catches it. ## Making the sum trustworthy 1. **Fail on an empty match.** Treat "no result files found" as a hard error in the collection step, never as a run with nothing wrong in it. 2. **Carry an expected count from outside the files.** The number of suites, or of tests, from the last known-good run or from the build's own inventory. The comparison has to come from somewhere the missing file could not have removed. 3. **Do not let the report replace the runner's own verdict.** Whether the test process exited successfully is an independent fact established outside these files; a green consolidation over an incomplete directory is not evidence of a green run. 4. **Log what the glob matched.** File count and total size in the job output turns a silent disappearance into something a person can see in a diff of two build logs. 5. **Alarm on a sudden drop in test count**, not only on failures. A large negative delta in `@tests` between consecutive runs is the cheapest available detector for this entire class of problem. ## The underlying point The per-file counters are a genuine convenience: a reader gets a suite summary out of one opening tag. What they are not is a description of the run. The run is the directory, the directory is a glob, and a glob has no cardinality of its own -- so any claim about the run as a whole is only as good as the out-of-band expectation you compare it against.
- What is the cheapest detector for a silently missing result file?A comparison against an expectation the missing file could not have removed: the number of result files or the total `@tests` from the last known-good run, or an inventory of suites the build knows it should have produced. Alarm on a sudden drop in test count, not only on failures - that catches the whole class of problem for almost no effort.
- Why is the runner's own exit status still worth checking when the consolidated report is green?Because it is established outside these files and survives their absence. A report summed over an incomplete directory can be green while the process that produced it died mid-run. Treat the report as a description of the results that arrived, and the exit status as the statement about whether the run finished at all.
saying these in an interview costs you the question
- Assumes one file holds the whole run's totals
- Treats a missing result file as a failed test
- Reads an empty glob as a run with no failures
- Trusts the consolidated report as proof the run finished
- Alarms only on failures, never on a falling test count