In a TestNG Selenium-Java suite, the @BeforeClass login hook throws and all 40 tests in that class report SKIPPED instead of FAILED; why, and how do you diagnose it?
answer
- precondition, not verdict
- configuration failure counted separately
- default policy is skip
- skipped result carries the hook's exception
- continue narrows, it does not rescue
basics
~20 sTestNG treats a failed configuration method as a broken precondition: the hook is reported as a configuration failure and every test depending on it is SKIPPED. Diagnosis starts at the hook's exception, which each skipped result also carries.
solid answer
~40 sIn TestNG a test only counts as FAILED when its own body fails; when a configuration method it depends on fails, the test never runs and is recorded as SKIPPED. With the default suite setting `configfailurepolicy="skip"`, a failed `@BeforeClass` marks that class's configuration as failed, so its `@BeforeMethod` hooks, all 40 test methods and its teardown hooks are skipped, except teardown marked `alwaysRun = true`. The console summary reads `Failures: 0, Skips: 40` with a separate `Configuration Failures: 1`. To diagnose, open the configuration failure entry, not a test; each skipped `ITestResult` also returns the hook's exception from `getThrowable()` and names the hook in `getSkipCausedBy()`. Switching to `continue` would not run these tests: in TestNG 7.12 it contains a failure to the narrowest scope, and here that scope is the whole class instance.
code
java · 23 linesimport org.testng.ITestResult;
import org.testng.annotations.AfterMethod;
import org.testng.annotations.BeforeClass;
import org.testng.annotations.Test;
public class CheckoutUiTest {
@BeforeClass
public void logIn() {
// Throws if the login page never loads; every test below is then SKIPPED.
throw new IllegalStateException("login page did not load within 30 s");
}
@Test
public void addsItemToCart() { }
@AfterMethod(alwaysRun = true)
public void explainSkips(ITestResult result) {
if (result.getStatus() == ITestResult.SKIP) {
System.out.println(result.getName() + " skipped because of " + result.getSkipCausedBy()
+ ": " + result.getThrowable());
}
}
}go deeper
Know the vocabulary: a failed setup hook is a configuration failure, and the tests that needed it are reported as skipped, not failed.
Explain the default skip policy, what each scope's failure takes down, and where the configuration failure shows up in the console summary and report.
Walk the diagnosis from the configuration failure to the root cause, use getThrowable() and getSkipCausedBy() on a skipped result, and explain why continue would not help a single-instance class.
Talk about treating skips as a pipeline signal and keeping failure-prone setup out of suite-wide scopes, since a single suite hook can silence a whole run.
## Why the tests are SKIPPED, not FAILED TestNG separates **test methods** (annotated `@Test`) from **configuration methods** (the `@Before*` and `@After*` hooks). A configuration method is a precondition. When it throws, TestNG reasons that the tests depending on it cannot produce a meaningful verdict, so it does not run them and records each one as **SKIPPED**. Only the hook itself is recorded as failed, and it is counted as a **configuration failure**, apart from test failures. For the scenario in the question, a UI suite logs in once per class in `@BeforeClass`. The login page times out, the hook throws, and the suite summary TestNG prints reads `Total tests run: 40, Passes: 0, Failures: 0, Skips: 40`, followed by a second line, `Configuration Failures: 1, Skips: ...`. That second line counts the failed hook and the class's other hooks, such as its `@BeforeMethod` calls and `@AfterClass`, that TestNG skipped because of it. Nothing is hidden: the failure is reported, just not where people first look. ## How far the cascade reaches The suite-level attribute `configfailurepolicy` on the `<suite>` element of `testng.xml` (also the `-configfailurepolicy` command-line option) accepts `skip` and `continue`. Its default is `skip`. What a failed hook takes down depends on its scope and on the policy, as described for TestNG 7.12: | Failed hook | `skip` (default) | `continue` | |---|---|---| | `@BeforeSuite` | every test in the suite is skipped | the same: a suite-level failure stops the suite | | `@BeforeTest` | every class in that `<test>` element is skipped | documented: the test methods of that `<test>` still run | | `@BeforeClass` | the class's tests and later hooks are skipped | only the failing instance's tests are skipped | | `@BeforeMethod` | the current test and the remaining tests of that class are skipped | only that one invocation is skipped; the hook is retried for the next test | Two points follow. First, a `@BeforeMethod` that fails once under `skip` takes every remaining test of the class with it, which surprises people who expected one skip. Second, `continue` does **not** turn skips back into runs here: the class has a single instance, its `@BeforeClass` failed, and that instance's 40 tests are still skipped. `continue` only helps when there are sibling invocations or instances, such as other test methods after a failed `@BeforeMethod`, or other instances created by a factory. ## Diagnosing it 1. **Read the configuration failure first.** In the HTML report or the console, the failed `@BeforeClass` method carries the real exception: a timeout, a wrong credential, an environment that is down. 2. **Confirm from any skipped test.** A skipped `ITestResult` returns the configuration method's exception from `getThrowable()`, and `getSkipCausedBy()` returns the upstream method or methods whose failure led to the skip. 3. **Check the scope.** Is the failing hook in this class, in a base class every UI class inherits, or at `@BeforeTest` or `@BeforeSuite` level? A base-class or suite-level hook explains why several classes all went to skips at once. 4. **Reproduce the hook alone.** The fix is in the precondition, for example the login flow or the test environment, not in the 40 tests. ## Keeping a skip cascade from hiding a broken run - **Do not treat skips as neutral.** A CI gate that fails only on test failures can show green for a run of 40 skips. TestNG's own command-line runner offers `-failwheneverythingskipped` for the all-skipped case, and its exit status sets a separate bit for skipped tests. - **Mark cleanup hooks `alwaysRun = true`.** Without it, a cascade also skips `@AfterMethod` and `@AfterClass`, so whatever the setup half-built stays behind. - **Use `continue` deliberately, not as a fix.** It narrows the blast radius of a flaky per-test setup step, but it never makes a broken setup green: the configuration failure is still reported, and a test whose own setup failed is still skipped. - **Keep heavy, failure-prone work out of wide scopes.** A login in `@BeforeSuite` that fails skips the entire suite under either policy. ## A note on the configuration hook timeout Configuration annotations accept `timeOut` in milliseconds. A hook that has not returned in time fails, and the tests depending on it are skipped in exactly the same way. A slow login page can therefore produce the same 40 skips as a thrown exception.
- In TestNG, if the same @BeforeMethod login fails before the first of 40 tests, how do the skip and continue policies differ?Under the default `skip`, the failure marks the class's configuration as failed, so the first test and the remaining tests of the class are skipped. Under `continue`, only that invocation is skipped; TestNG re-runs `@BeforeMethod` before the next test, and tests whose setup then succeeds run normally.
- Why does a TestNG @BeforeSuite failure skip the whole run even with configfailurepolicy set to continue?A suite-level configuration failure marks the whole suite as failed. TestNG then skips every remaining test method, and every configuration method not marked `alwaysRun = true`, whatever the policy, because no narrower scope exists to contain the failure.
- How can a TestNG run that skipped every test still show up green, and what does TestNG itself offer against that?Skips are not failures, so a gate that only counts failures passes a run of zero passes and many skips. TestNG's command-line runner has `-failwheneverythingskipped` to fail an all-skipped run, and its exit status flags skipped tests separately, so a pipeline can treat skips as a signal.
- A failed @BeforeClass marks every test in the class as FAILED.
- Setting configfailurepolicy to continue makes the 40 skipped tests run.
- Skipped tests mean the tests themselves are broken and need fixing.
- A skipped TestNG test has no record of why it was skipped.
- A class's @AfterClass still runs after its @BeforeClass fails, even without alwaysRun.
- configfailurepolicy=continue hides the configuration failure from the report.
Assumes TestNG 7.12; the per-scope behaviour of configfailurepolicy and ITestResult.getSkipCausedBy() are as in the 7.12 source.