skip to content

After a failing TestNG cross-browser run, what does testng-failed.xml contain, and why can a green rerun of it still hide the failure's real cause?

level: seniorimportance: should knowfreq 45%

answer

  1. written by a default reporter
  2. failed plus skipped, plus their prerequisites
  3. same attributes, far fewer methods
  4. missing neighbours, missing contention

basics

~20 s

testng-failed.xml is a generated suite holding the failed and skipped test methods, the methods they depend on and their configuration methods, with the original attributes and parameters. A green rerun runs far fewer tests, so interference and load disappear.

solid answer

~50 s

When a run has failed or skipped tests, TestNG's default reporter writes `testng-failed.xml` to the output directory (`test-output` by default). It copies the suite's attributes, including `parallel`, `thread-count` and suite parameters, and for each `<test>` that had failures or skips adds a `<test>` named with a `(failed)` suffix that keeps that `<test>`'s parameters and parallel mode. It includes the failed and skipped test methods, the test methods they depend on, and the relevant configuration methods; data-driven methods carry `invocation-numbers` for just the failing rows. A green rerun is weak evidence: the methods that ran beside the failures, and may have polluted shared state or loaded the environment, are absent, and far fewer methods compete for the pool and the browsers. Treat it as a clue that the failure depends on context, then reproduce it in the full run.

code

xml · 14 lines
xml
<suite name="Failed suite [cross-browser]" parallel="tests" thread-count="2">
  <test name="firefox(failed)">
    <parameter name="browser" value="firefox"/>
    <classes>
      <class name="com.shop.CheckoutTest">
        <methods>
          <include name="openBrowser"/>
          <include name="payWithCard"/>
          <include name="searchProducts" invocation-numbers="2"/>
        </methods>
      </class>
    </classes>
  </test>
</suite>

go deeper

for a junior

Know that TestNG writes testng-failed.xml to the output directory after failures and that you can run it like any other suite file.

for a middle

Explain what goes in: failed and skipped methods, their dependencies and configuration, the original attributes and per-test parameters, invocation numbers for data rows.

for a senior

Argue why a green rerun is weak evidence: missing neighbours, lower contention, new ordering and skips turning into runs. Describe how you would reproduce in the full run.

for a principal

Set the team rule for how reruns are reported, so a rerun pass never silently replaces a first-run failure in the record.

## Where testng-failed.xml comes from `testng-failed.xml` is produced by TestNG's built-in `FailedReporter`, one of the **default reporters** TestNG registers unless default listeners are switched off. After each suite finishes, it looks at every `<test>` element's results. If a `<test>` had **any failed or any skipped test method**, it contributes to a new suite file; if nothing failed or was skipped anywhere, no file is written. The file goes to the output directory, `test-output` by default, and to the suite's own output folder. Rerunning it is simply running another suite file: ```bash java org.testng.TestNG -d test-outputs test-outputs/testng-failed.xml ``` ## What the generated suite contains Read against the 7.12 source, the generated file is built like this: 1. **A shallow copy of the original suite**, renamed `Failed suite [<original name>]`, keeping `parallel`, `thread-count`, `data-provider-thread-count`, `configfailurepolicy`, the suite-level parameters, listeners and packages. 2. **One `<test>` per original `<test>` that had failures or skips**, named `<original name>(failed)`. It keeps that `<test>`'s own parameters, its included and excluded groups, and its parallel mode, so the `browser` value of a per-browser `<test>` survives. 3. **The methods to rerun**: every failed or skipped `@Test` method, plus the test methods each one depends on (the transitive `dependsOn` chain), so a test skipped because its upstream failed can run again. 4. **Configuration methods**: the configuration methods of the classes involved, at suite, test, class and method level, group configuration where relevant, and any suite-, test- or group-level configuration method that itself failed or was skipped. 5. **Per-row selection for repeated methods**: for a data-driven or `invocationCount` method, each `<include>` lists `invocation-numbers` of the failing iterations, and an iteration whose identical invocation also passed is left out. | In the original run | In testng-failed.xml | |---|---| | a failed test method | included | | a test skipped because its upstream failed | included, with its upstream | | a passed test that no failed or skipped test depends on | absent | | row 3 of a data provider failed | that method, `invocation-numbers` naming row 3 | | the suite's `parallel` and `thread-count` | copied unchanged | ## Why a green rerun can mislead The rerun keeps the configuration but changes the **population**. That is exactly where most intermittent suite failures live: - **Missing neighbours.** Tests that passed are gone. If one of them left a record, a cookie, a logged-in session or a static value behind that broke a later test, the culprit is no longer in the run, and the failure vanishes. - **Less contention.** With `parallel="methods"` and `thread-count="8"`, the full run kept eight methods busy against one test environment; a rerun of four methods never reaches that load, so timeouts and race conditions do not reproduce. - **Different ordering.** The rerun's methods sit in a new, much smaller schedule. Order-dependent behaviour, including a method that only passes when another happened to run first on the same thread, changes with it. - **Skips that become runs.** Tests reported as **skipped** in the first run, because a configuration method or a dependency failed, now execute. A green result for them says nothing about the original cause of the skip. - **Environment drift.** Minutes later the backend, the data and the browser node may simply be healthier. ## How to use it well - Use the rerun to **separate deterministic failures from context-dependent ones**: a test that still fails alone is a direct defect to fix first. - For tests that pass alone, **reproduce in the full suite**: run the original file again, or narrow it while keeping the same `parallel` mode and `thread-count`, to find which neighbour or which load triggers the failure. - **Never count a green rerun as a pass for the original run.** Report both results; the first run's failures stand until explained. - **Open the generated file before rerunning**: the `(failed)` test names, parameters and `invocation-numbers` show exactly what will run. The reporter writes the file only when a run has failures or skips and does not itself delete an older copy, so check that it came from the run you are investigating. - Check whether the original failures were **failed** or **skipped**: a skip cascade points at a configuration method or a dependency, not at the skipped test. Isolating the shared state once found is a harness-design question; the TestNG-specific point is that `testng-failed.xml` reproduces the suite's settings, not its circumstances.

  • Why does a test that was SKIPPED in the first TestNG run appear in testng-failed.xml?
    The reporter collects skipped as well as failed test methods, because a skip usually means a configuration method or an upstream dependency failed. It adds the methods the skipped test depends on, so the rerun can execute it rather than skipping it again.
  • A TestNG data-driven test failed on rows 2 and 5 of ten. What does testng-failed.xml rerun?
    Only those iterations. The method's `<include>` carries `invocation-numbers` listing the failing indices, so the data provider is still called but only rows 2 and 5 are invoked. A row whose identical invocation also passed in the run is treated as flaky and left out.

  • Counts a green testng-failed.xml rerun as proof that the original failures are fixed
  • Believes the generated file drops the original parallel mode and parameters
  • Thinks only failed methods are included, never skipped ones or their dependencies
  • Assumes a skipped test in the first run failed its own assertions
  • Expects testng-failed.xml to be written even when every test passed

Assumes TestNG 7.12; what the failed-suite reporter copies and selects was checked against its 7.12.0 source.