skip to content

In a TestNG Selenium-Java suite, the @BeforeClass login hook throws and all 40 tests in that class report SKIPPED instead of FAILED; why, and how do you diagnose it?

level: seniorimportance: must knowfreq 55%

answer

  1. precondition, not verdict
  2. configuration failure counted separately
  3. default policy is skip
  4. skipped result carries the hook's exception
  5. continue narrows, it does not rescue

basics

~20 s

TestNG treats a failed configuration method as a broken precondition: the hook is reported as a configuration failure and every test depending on it is SKIPPED. Diagnosis starts at the hook's exception, which each skipped result also carries.

solid answer

~40 s

In TestNG a test only counts as FAILED when its own body fails; when a configuration method it depends on fails, the test never runs and is recorded as SKIPPED. With the default suite setting `configfailurepolicy="skip"`, a failed `@BeforeClass` marks that class's configuration as failed, so its `@BeforeMethod` hooks, all 40 test methods and its teardown hooks are skipped, except teardown marked `alwaysRun = true`. The console summary reads `Failures: 0, Skips: 40` with a separate `Configuration Failures: 1`. To diagnose, open the configuration failure entry, not a test; each skipped `ITestResult` also returns the hook's exception from `getThrowable()` and names the hook in `getSkipCausedBy()`. Switching to `continue` would not run these tests: in TestNG 7.12 it contains a failure to the narrowest scope, and here that scope is the whole class instance.

code

java · 23 lines
java
import org.testng.ITestResult;
import org.testng.annotations.AfterMethod;
import org.testng.annotations.BeforeClass;
import org.testng.annotations.Test;

public class CheckoutUiTest {
  @BeforeClass
  public void logIn() {
    // Throws if the login page never loads; every test below is then SKIPPED.
    throw new IllegalStateException("login page did not load within 30 s");
  }

  @Test
  public void addsItemToCart() { }

  @AfterMethod(alwaysRun = true)
  public void explainSkips(ITestResult result) {
    if (result.getStatus() == ITestResult.SKIP) {
      System.out.println(result.getName() + " skipped because of " + result.getSkipCausedBy()
          + ": " + result.getThrowable());
    }
  }
}

go deeper

for a junior

Know the vocabulary: a failed setup hook is a configuration failure, and the tests that needed it are reported as skipped, not failed.

for a middle

Explain the default skip policy, what each scope's failure takes down, and where the configuration failure shows up in the console summary and report.

for a senior

Walk the diagnosis from the configuration failure to the root cause, use getThrowable() and getSkipCausedBy() on a skipped result, and explain why continue would not help a single-instance class.

for a principal

Talk about treating skips as a pipeline signal and keeping failure-prone setup out of suite-wide scopes, since a single suite hook can silence a whole run.

## Why the tests are SKIPPED, not FAILED TestNG separates **test methods** (annotated `@Test`) from **configuration methods** (the `@Before*` and `@After*` hooks). A configuration method is a precondition. When it throws, TestNG reasons that the tests depending on it cannot produce a meaningful verdict, so it does not run them and records each one as **SKIPPED**. Only the hook itself is recorded as failed, and it is counted as a **configuration failure**, apart from test failures. For the scenario in the question, a UI suite logs in once per class in `@BeforeClass`. The login page times out, the hook throws, and the suite summary TestNG prints reads `Total tests run: 40, Passes: 0, Failures: 0, Skips: 40`, followed by a second line, `Configuration Failures: 1, Skips: ...`. That second line counts the failed hook and the class's other hooks, such as its `@BeforeMethod` calls and `@AfterClass`, that TestNG skipped because of it. Nothing is hidden: the failure is reported, just not where people first look. ## How far the cascade reaches The suite-level attribute `configfailurepolicy` on the `<suite>` element of `testng.xml` (also the `-configfailurepolicy` command-line option) accepts `skip` and `continue`. Its default is `skip`. What a failed hook takes down depends on its scope and on the policy, as described for TestNG 7.12: | Failed hook | `skip` (default) | `continue` | |---|---|---| | `@BeforeSuite` | every test in the suite is skipped | the same: a suite-level failure stops the suite | | `@BeforeTest` | every class in that `<test>` element is skipped | documented: the test methods of that `<test>` still run | | `@BeforeClass` | the class's tests and later hooks are skipped | only the failing instance's tests are skipped | | `@BeforeMethod` | the current test and the remaining tests of that class are skipped | only that one invocation is skipped; the hook is retried for the next test | Two points follow. First, a `@BeforeMethod` that fails once under `skip` takes every remaining test of the class with it, which surprises people who expected one skip. Second, `continue` does **not** turn skips back into runs here: the class has a single instance, its `@BeforeClass` failed, and that instance's 40 tests are still skipped. `continue` only helps when there are sibling invocations or instances, such as other test methods after a failed `@BeforeMethod`, or other instances created by a factory. ## Diagnosing it 1. **Read the configuration failure first.** In the HTML report or the console, the failed `@BeforeClass` method carries the real exception: a timeout, a wrong credential, an environment that is down. 2. **Confirm from any skipped test.** A skipped `ITestResult` returns the configuration method's exception from `getThrowable()`, and `getSkipCausedBy()` returns the upstream method or methods whose failure led to the skip. 3. **Check the scope.** Is the failing hook in this class, in a base class every UI class inherits, or at `@BeforeTest` or `@BeforeSuite` level? A base-class or suite-level hook explains why several classes all went to skips at once. 4. **Reproduce the hook alone.** The fix is in the precondition, for example the login flow or the test environment, not in the 40 tests. ## Keeping a skip cascade from hiding a broken run - **Do not treat skips as neutral.** A CI gate that fails only on test failures can show green for a run of 40 skips. TestNG's own command-line runner offers `-failwheneverythingskipped` for the all-skipped case, and its exit status sets a separate bit for skipped tests. - **Mark cleanup hooks `alwaysRun = true`.** Without it, a cascade also skips `@AfterMethod` and `@AfterClass`, so whatever the setup half-built stays behind. - **Use `continue` deliberately, not as a fix.** It narrows the blast radius of a flaky per-test setup step, but it never makes a broken setup green: the configuration failure is still reported, and a test whose own setup failed is still skipped. - **Keep heavy, failure-prone work out of wide scopes.** A login in `@BeforeSuite` that fails skips the entire suite under either policy. ## A note on the configuration hook timeout Configuration annotations accept `timeOut` in milliseconds. A hook that has not returned in time fails, and the tests depending on it are skipped in exactly the same way. A slow login page can therefore produce the same 40 skips as a thrown exception.

  • In TestNG, if the same @BeforeMethod login fails before the first of 40 tests, how do the skip and continue policies differ?
    Under the default `skip`, the failure marks the class's configuration as failed, so the first test and the remaining tests of the class are skipped. Under `continue`, only that invocation is skipped; TestNG re-runs `@BeforeMethod` before the next test, and tests whose setup then succeeds run normally.
  • Why does a TestNG @BeforeSuite failure skip the whole run even with configfailurepolicy set to continue?
    A suite-level configuration failure marks the whole suite as failed. TestNG then skips every remaining test method, and every configuration method not marked `alwaysRun = true`, whatever the policy, because no narrower scope exists to contain the failure.
  • How can a TestNG run that skipped every test still show up green, and what does TestNG itself offer against that?
    Skips are not failures, so a gate that only counts failures passes a run of zero passes and many skips. TestNG's command-line runner has `-failwheneverythingskipped` to fail an all-skipped run, and its exit status flags skipped tests separately, so a pipeline can treat skips as a signal.

  • A failed @BeforeClass marks every test in the class as FAILED.
  • Setting configfailurepolicy to continue makes the 40 skipped tests run.
  • Skipped tests mean the tests themselves are broken and need fixing.
  • A skipped TestNG test has no record of why it was skipped.
  • A class's @AfterClass still runs after its @BeforeClass fails, even without alwaysRun.
  • configfailurepolicy=continue hides the configuration failure from the report.

Assumes TestNG 7.12; the per-scope behaviour of configfailurepolicy and ITestResult.getSkipCausedBy() are as in the 7.12 source.