When JUnit 5's `LegacyXmlReportGeneratingListener` writes a failed test case to JUnit-XML, what decides whether it emits `<failure>` or `<error>`, and why does the choice matter to whatever reads the file?

middlemust knowfreq 66%

answer

the thrown object decides
assertion versus everything else
instanceof AssertionError
one reader maps them FAILED and BROKEN

basics

~20 s

The thrown object decides. An AssertionError becomes <failure>; any other throwable becomes <error>. The split matters because consumers bucket them differently -- Allure 2's JUnit-XML reader records FAILED for one and BROKEN for the other.

solid answer

~40 s

The listener classifies each failed result with a single type test. Its internal classification has the members SUCCESS, SKIPPED, FAILURE and ERROR, and a failed result is FAILURE when its throwable is an instance of AssertionError and ERROR otherwise. So an assertion that did not hold writes <failure>, while a missing file or a null dereference writes <error>. Because the test is instanceof, subclasses that assertion libraries throw still count as failures, but an assertion caught and rewrapped in a RuntimeException becomes an error. Downstream that distinction is not cosmetic: Allure 2's JunitXmlPlugin checks for <failure> first and records FAILED, and records BROKEN for <error>, so the same red case lands in a different bucket.

The listener classifies each failed result with a single type test. Its internal classification has the members `SUCCESS`, `SKIPPED`, `FAILURE` and `ERROR`, and a failed result is `FAILURE` when its throwable is an instance of `AssertionError` and `ERROR` otherwise. So an assertion that did not hold writes `<failure>`, while a missing file or a null dereference writes `<error>`. Because the test is `instanceof`, subclasses that assertion libraries throw still count as failures, but an assertion caught and rewrapped in a `RuntimeException` becomes an error. Downstream that distinction is not cosmetic: Allure 2's `JunitXmlPlugin` checks for `<failure>` first and records `FAILED`, and records `BROKEN` for `<error>`, so the same red case lands in a different bucket.

code

xml · 10 lines

xml

<testsuite name="com.example.AuthTest" tests="2" errors="1" skipped="0" failures="1" time="0.016">
  <testcase name="rejectsExpiredToken" classname="com.example.AuthTest" time="0.014">
    <failure message="token was accepted" type="java.lang.AssertionError">java.lang.AssertionError: token was accepted
	at com.example.AuthTest.rejectsExpiredToken(AuthTest.java:57)</failure>
  </testcase>
  <testcase name="loadsKeystore" classname="com.example.AuthTest" time="0.002">
    <error message="keystore not found" type="java.io.FileNotFoundException">java.io.FileNotFoundException: keystore not found
	at com.example.AuthTest.loadsKeystore(AuthTest.java:23)</error>
  </testcase>
</testsuite>

go deeper

for a junior

Know that both elements mean the case is red, and that <failure> means an assertion did not hold while <error> means the case ended on something that was not an assertion.

for a middle

Explain the actual test the writer applies -- is the throwable an AssertionError -- and why that makes subclasses count as failures while a rewrapped assertion becomes an error.

for a senior

Show you have seen the downstream effect: a helper that starts wrapping throwables can move an entire suite's red cases between a reader's buckets with no behaviour change at all.

for a principal

Be ready to argue whether your organisation's reports should preserve the failure-versus-error split or normalise it away, and what triage capability each choice buys or discards.

Two elements for one colour

<failure> and <error> both mean the case is red. The Maven Surefire report schema declares them almost identically: both are simple-content elements whose text is the stack trace, and both carry the optional attributes @message and @type. The difference is entirely semantic, and it is the oldest distinction in this file format:

<failure> -- the test said no. An assertion the test itself wrote did not hold.
<error> -- something else went wrong. The test never reached a verdict, because a throwable that was not an assertion escaped.

What decides which one is written

JUnit 5's LegacyXmlReportGeneratingListener makes the decision with a single type test. Its internal classification has four members -- SUCCESS, SKIPPED, FAILURE and ERROR -- and for a result whose status is failed it asks whether the recorded throwable is an instance of AssertionError. If it is, the case is classified FAILURE and <failure> is written. If it is anything else -- or if no throwable was recorded at all -- the case is classified ERROR and <error> is written.

Three consequences follow directly from that being an instanceof check:

Subclasses count. Assertion libraries throw their own types that extend AssertionError, and every one of those still lands in <failure>.
Wrapping breaks it. An assertion failure caught and rethrown inside a RuntimeException is no longer an AssertionError, so it is written as <error>. The test's intent is unchanged; its representation in the file is not.
A library that does not extend AssertionError is invisible to the rule. A custom check that throws a plain IllegalStateException on mismatch produces <error> for what its author meant as a failed assertion.

Maven Surefire carries the same distinction differently: it does not re-derive it from the throwable at write time. Its ReportEntryType enum has separate ERROR and FAILURE rows, mapping to the tags error and failure, and the provider that actually ran the tests decides which row a given result gets.

Why the choice matters downstream

Both elements make the case red, so a reader that only asks "is this case green?" cannot tell them apart. Every reader that does more than that treats them as different things.

	`<failure>`	`<error>`
written when	the throwable is an `AssertionError`	the throwable is anything else
meaning	the assertion did not hold	the case could not reach a verdict
counted in	the suite's failure count	the suite's error count
Allure 2's `JunitXmlPlugin` records	`FAILED`	`BROKEN`

That last row is the one that bites. Allure 2's JUnit-XML reader checks for <failure> first and records FAILED; if there is no <failure> but there is an <error>, it records BROKEN. Those are separate buckets in the generated report, so the same red case appears in a different place depending only on which element the writer chose. A build whose assertion helper starts wrapping its throwables can move its entire red set from one bucket to the other without a single test changing behaviour.

What the split is genuinely useful for

The reason the distinction survived decades of this format is that the two outcomes want different responses:

A <failure> usually means the product under test behaved differently from what the test asserted. Someone reads the assertion and decides whether the test or the product is wrong.
An <error> usually means the test's own scaffolding did not hold up -- a missing file, an unreachable dependency, a null where the fixture promised an object. Reading the assertion tells you nothing, because no assertion was reached.

Sorting a wall of red into those two piles is often the fastest triage a report can offer, and it costs nothing at read time, because the writer already made the call.

Getting it right in your own code

Let assertion libraries throw AssertionError subclasses; do not catch and rewrap them just to add context, or every failure becomes an error.
If you write your own check helper, extend AssertionError so it is classified as a failure.
When reading these files, do not collapse the two elements into one red bucket by default -- that discards the only triage signal the format offers for free.
Do not over-read it either: a case with <error> is not necessarily an infrastructure problem. It tells you only that the thrown object was not an assertion.

## Two elements for one colour `<failure>` and `<error>` both mean the case is red. The Maven Surefire report schema declares them almost identically: both are simple-content elements whose text is the stack trace, and both carry the optional attributes `@message` and `@type`. The difference is entirely semantic, and it is the oldest distinction in this file format: - **`<failure>`** -- the test said *no*. An assertion the test itself wrote did not hold. - **`<error>`** -- something else went wrong. The test never reached a verdict, because a throwable that was not an assertion escaped. ## What decides which one is written JUnit 5's `LegacyXmlReportGeneratingListener` makes the decision with a single type test. Its internal classification has four members -- `SUCCESS`, `SKIPPED`, `FAILURE` and `ERROR` -- and for a result whose status is failed it asks whether the recorded throwable is an instance of `AssertionError`. If it is, the case is classified `FAILURE` and `<failure>` is written. If it is anything else -- or if no throwable was recorded at all -- the case is classified `ERROR` and `<error>` is written. Three consequences follow directly from that being an `instanceof` check: 1. **Subclasses count.** Assertion libraries throw their own types that extend `AssertionError`, and every one of those still lands in `<failure>`. 2. **Wrapping breaks it.** An assertion failure caught and rethrown inside a `RuntimeException` is no longer an `AssertionError`, so it is written as `<error>`. The test's intent is unchanged; its representation in the file is not. 3. **A library that does not extend `AssertionError` is invisible to the rule.** A custom check that throws a plain `IllegalStateException` on mismatch produces `<error>` for what its author meant as a failed assertion. Maven Surefire carries the same distinction differently: it does not re-derive it from the throwable at write time. Its `ReportEntryType` enum has separate `ERROR` and `FAILURE` rows, mapping to the tags `error` and `failure`, and the provider that actually ran the tests decides which row a given result gets. ## Why the choice matters downstream Both elements make the case red, so a reader that only asks "is this case green?" cannot tell them apart. Every reader that does more than that treats them as different things. | | `<failure>` | `<error>` | |---|---|---| | written when | the throwable is an `AssertionError` | the throwable is anything else | | meaning | the assertion did not hold | the case could not reach a verdict | | counted in | the suite's failure count | the suite's error count | | Allure 2's `JunitXmlPlugin` records | `FAILED` | `BROKEN` | That last row is the one that bites. Allure 2's JUnit-XML reader checks for `<failure>` first and records `FAILED`; if there is no `<failure>` but there is an `<error>`, it records `BROKEN`. Those are separate buckets in the generated report, so the same red case appears in a different place depending only on which element the writer chose. A build whose assertion helper starts wrapping its throwables can move its entire red set from one bucket to the other without a single test changing behaviour. ## What the split is genuinely useful for The reason the distinction survived decades of this format is that the two outcomes want different responses: - A `<failure>` usually means the product under test behaved differently from what the test asserted. Someone reads the assertion and decides whether the test or the product is wrong. - An `<error>` usually means the test's own scaffolding did not hold up -- a missing file, an unreachable dependency, a null where the fixture promised an object. Reading the assertion tells you nothing, because no assertion was reached. Sorting a wall of red into those two piles is often the fastest triage a report can offer, and it costs nothing at read time, because the writer already made the call. ## Getting it right in your own code - Let assertion libraries throw `AssertionError` subclasses; do not catch and rewrap them just to add context, or every failure becomes an error. - If you write your own check helper, extend `AssertionError` so it is classified as a failure. - When reading these files, do not collapse the two elements into one red bucket by default -- that discards the only triage signal the format offers for free. - Do not over-read it either: a case with `<error>` is not necessarily an infrastructure problem. It tells you only that the thrown object was not an assertion.

A team catches assertion failures and rethrows them inside a custom RuntimeException for extra context. What happens to their JUnit-XML?
Every one of those cases is written as `<error>` instead of `<failure>`, because the classification tests the thrown object for `AssertionError` and a wrapper is not one. The counts move from the failure column to the error column, and a reader such as Allure 2 reclassifies the whole set from `FAILED` to `BROKEN`.
Does `<skipped>` take the same attributes as `<failure>` and `<error>`?
No. The Surefire schema declares `@message` on `<skipped>`, but `@message` and `@type` on `<failure>` and `<error>`. A skip has no exception type to name, so the schema gives it nowhere to put one -- worth knowing before you write a reader that expects `@type` on every outcome element.

saying these in an interview costs you the question

Says <failure> and <error> are two names for the same thing
Claims the assertion library chooses the element, not the thrown type
Thinks only java.lang.AssertionError itself counts, never a subclass
Assumes every consumer collapses both into one red bucket

When JUnit 5's `LegacyXmlReportGeneratingListener` writes a failed test case to JUnit-XML, what decides whether it emits `<failure>` or `<error>`, and why does the choice matter to whatever reads the file?

§ answer

basics

solid answer

code

§ go deeper

▾level notes4 levels

▾deep dive~4 min

Two elements for one colour

What decides which one is written

Why the choice matters downstream

What the split is genuinely useful for

Getting it right in your own code

▾common follow-ups2 questions

▾red flags4 flags

Two elements for one colour

What decides which one is written

Why the choice matters downstream

What the split is genuinely useful for

Getting it right in your own code

answer

go deeper