skip to content

Testsuite and Testcase

The shape of the file itself, and its one surprise: the Surefire schema declares no <testsuites> root, so the aggregating wrapper every CI consumer accepts is a dialect nobody wrote down.

on this pageshow

explore

questions

5

In a JUnit-XML result file written to the Maven Surefire report schema, which attributes does a `<testcase>` element require, which are optional, and what does a reader have to work with when it needs to tell two cases apart?

level: juniorimportance: must knowfreq 56%

answer

  1. two attributes, one of them a duration
  2. the class half is optional
  3. a bare name is not unique
  4. identity is @classname plus @name
  5. no id attribute exists

basics

~20 s

A <testcase> requires only @name and @time; @classname, @group and @timestamp are optional. The identity the file declares is therefore @classname plus @name - and a writer may legally omit the @classname half of that pair.

solid answer

~40 s

The schema requires `@name` and `@time` on `<testcase>`; `@classname`, `@group` and `@timestamp` are optional. That is the whole vocabulary the file offers for saying which test a row represents - there is no id attribute and no ordinal. `@name` alone identifies nothing, because short test names repeat across classes, so the pair consumers key on is `@classname` plus `@name`. The uncomfortable part is that only half of that pair is guaranteed: a conforming file can present cases carrying nothing but a name and a duration, and a reader keying on the pair then has to invent a fallback. Nor is the pair unique - the schema declares no uniqueness constraint, so two identical `@classname` and `@name` combinations in one file are valid.

code

xml · 4 lines
xml
<testsuite name="orders" tests="2" errors="0" skipped="0" failures="0">
  <testcase name="createsOrder" classname="com.example.OrderTest" time="0.031"/>
  <testcase name="createsOrder" classname="com.example.LegacyOrderTest" time="0.044"/>
</testsuite>

go deeper

for a junior

Recall that only @name and @time are required on a <testcase>, that @classname is optional, and that the two together are what a reader uses to name a case.

for a middle

Explain why a bare @name is ambiguous across classes, and what a consumer must do when the optional @classname is missing from an otherwise conforming file.

for a senior

Describe the reporting damage you have actually seen: unrelated cases merged under one name, or duplicate pairs silently overwriting each other in a results store.

for a principal

Decide what your organisation keys results on, given that the file guarantees only half an identifier, and who owns that derivation so every consumer computes the same key.

## What a case element must declare In the Maven Surefire report schema, a `<testcase>` element is required to carry exactly two attributes: `@name` and `@time`. `@classname`, `@group` and `@timestamp` are optional. That is the entire attribute vocabulary. There is no id attribute, no ordinal, no slot a writer can use to carry a key of its own invention. Whatever a consumer wants to know about *which* test a row represents, it has to reconstruct from what is there. ## The identity the file actually declares `@name` on its own identifies nothing. Short test names repeat across a codebase, and a single suite can easily contain several `<testcase>` elements sharing one `@name`. The identifying pair the file offers is therefore `@classname` plus `@name`, and that pair is what nearly every consumer keys on when it matches a case in this run against the same case in the last one. The awkward part is that only half of that pair is guaranteed: | attribute | required? | role in identity | |---|---|---| | `@name` | yes | the case's own name | | `@time` | yes | its duration -- nothing to do with identity | | `@classname` | no | the qualifier that makes `@name` unambiguous | | `@group`, `@timestamp` | no | extra context, not identity | ## When the optional half is missing A conforming file may present cases carrying nothing but `@name` and `@time`. Consumers cope with that in incompatible ways, none of which the format prescribes: - bucket every unqualified case under an empty or placeholder class name; - fall back to the enclosing `<testsuite>`'s `@name` as the qualifier; - key on `@name` alone and silently merge unrelated cases that happen to share it. If your reporting stack has ever shown one case with an implausible run count, this is a strong candidate for why. ## No uniqueness constraint anywhere The schema declares no uniqueness rule on either attribute. Two `<testcase>` elements with the same `@classname` and the same `@name` in one file are perfectly valid, and a test executed more than once under one name produces exactly that. Again the consumers diverge: some keep both rows, some let the later one overwrite the earlier, some append an index to make the pair unique. The file contains nothing that says which is correct, so the same artefact can legitimately produce different case counts in different tools. ## Reading a case element defensively 1. **Treat `@classname` as optional in code**, not merely in theory -- a null-safe fallback beats a crash on the one suite whose writer omits it. 2. **Never assume the pair is unique.** Aggregate by it; do not index by it, unless you have verified the writer feeding your own pipeline. 3. **Do not use document order as identity.** Nothing in the schema fixes the order of `<testcase>` elements, so position is not stable between runs. 4. **Do not invent an id and hide it.** If your store needs a durable key, derive it explicitly and record how you derived it, so a later reader can reproduce the mapping. ## The inversion worth noticing The format guarantees you a duration for every single case and does not guarantee you a class name for any of them. That tells you what the file was designed to do: fill in a summary table of names and timings. Identity was an afterthought, and every tool that consolidates these files is still paying for it.

  • Two `<testcase>` elements in one file carry the same `@classname` and the same `@name`. Is the file valid, and what do consumers do with it?
    It is valid - the schema declares no uniqueness constraint on either attribute, and a test executed more than once under one name produces exactly that shape. Consumers diverge: some keep both rows, some let the later overwrite the earlier, some append an index. The file says nothing about which is right.
  • If `@classname` is absent, what can a reader legitimately fall back on?
    Only what is in the file: the enclosing `<testsuite>`'s `@name`, or nothing at all. Both are conventions rather than rules - the schema neither supplies a default nor says the suite name qualifies the case. Whatever you choose, record it, because a different tool reading the same file will choose differently.

saying these in an interview costs you the question

  • Thinks @classname is required on every <testcase>
  • Treats @name alone as a unique case identifier
  • Expects an id attribute the schema never declares
  • Assumes document order is a stable case identity
  • Believes the schema forbids duplicate name pairs
open as a page

In the Maven Surefire report schema (`surefire-test-report.xsd`), which attributes must every `<testsuite>` element carry, and what does that schema actually guarantee about how those counters relate to the `<testcase>` elements inside it?

level: middleimportance: must knowfreq 62%

basics

~20 s

Every <testsuite> must carry @name, @tests, @errors, @skipped and @failures; @time and @timestamp are optional. The schema only requires those attributes to be present - it never makes their values agree with the <testcase> elements below.

open as a page

In the Maven Surefire report schema, what does the `<properties>` block inside a `<testsuite>` hold, and why can a fact about one individual `<testcase>` never be recorded there?

level: middleimportance: should knowfreq 38%

basics

~20 s

<properties> holds a flat list of <property> elements, each carrying a @name and a @value attribute. It is a child of <testsuite>, so everything in it describes the whole file - <testcase> has no properties block of its own.

open as a page

Maven Surefire writes one JUnit-XML file per test class, named `TEST-<sourceName>.xml`, and each file's `<testsuite>` carries its own `@tests`, `@failures`, `@errors` and `@skipped`. Where does a run-level total come from, and what happens to it when one of those files is never written?

level: seniorimportance: should knowfreq 47%

basics

~20 s

No file holds a run total. Each TEST-<sourceName>.xml declares counters for its own suite only, so the run figure is a consumer's sum over the glob - and a file that was never written drops out of that sum silently.

open as a page

The Maven Surefire report schema declares `@time` and `@timestamp` on `<testsuite>` as optional, while `@time` on `<testcase>` is required. What may a consumer assume about a suite's duration and start time, and how should it behave when they are absent?

level: middleimportance: nice to knowfreq 29%

basics

~20 s

Nothing is guaranteed at suite level: @time and @timestamp are optional, so a valid file may carry neither. Only <testcase> @time is required, so a reader can sum case durations - which is a different quantity.

open as a page