skip to content

TestNG

The other JVM runner: groups, data providers, declared dependencies between tests, and an XML suite file that configures parallelism. Asked so you can justify it against JUnit for a given suite.

on this pageshow

explore

questions

page 1 of 2

In TestNG, how does a hard assert from org.testng.Assert differ from a SoftAssert, and when should you use each?

level: juniorimportance: must knowfreq 55%

answer

  1. stop at the first failure, or not
  2. AssertionError thrown on the spot
  3. failures recorded, verdict deferred
  4. assertAll() delivers the verdict

basics

~20 s

A hard assert from org.testng.Assert throws an AssertionError at the first failed check and ends the test method. A SoftAssert records each failure and keeps going; only its assertAll() call throws, listing every recorded failure.

solid answer

~40 s

The static methods of `org.testng.Assert` are **hard**: the first failing check throws `AssertionError`, the rest of the method never runs, and TestNG marks the test failed. `org.testng.asserts.SoftAssert` is an object you create: each `softly.assertEquals(...)` that fails is recorded instead of thrown, and nothing fails until you call `softly.assertAll()`, which throws one `AssertionError` listing every recorded failure. I use hard asserts for preconditions, because if the page or response did not load, every later check is noise, and soft asserts when one test verifies several independent fields, so a single run shows every mismatch. The classic trap is forgetting `assertAll()`: the failures are recorded but never thrown, and the test passes.

code

java · 24 lines
java
import org.testng.Assert;
import org.testng.annotations.Test;
import org.testng.asserts.SoftAssert;

public class ProfileHeaderTest {

  record Profile(String name, String email, boolean verified) {}

  @Test
  public void headerShowsEveryField() {
    Profile profile = loadProfile("u-42");
    Assert.assertNotNull(profile, "profile loaded");

    SoftAssert softly = new SoftAssert();
    softly.assertEquals(profile.name(), "Ada Lovelace", "name");
    softly.assertEquals(profile.email(), "[email protected]", "email");
    softly.assertTrue(profile.verified(), "verified badge");
    softly.assertAll();
  }

  private Profile loadProfile(String id) {
    return new Profile("Ada Lovelace", "[email protected]", true);
  }
}

go deeper

for a junior

Recall that org.testng.Assert stops the method at the first failed check, while a SoftAssert records failures until assertAll() throws them together.

for a middle

Explain that SoftAssert is an instance you create per test method, that it intercepts only AssertionError from its own checks, and that assertAll() must be the last statement.

for a senior

Show judgment about mixing the styles: hard guards for preconditions, soft checks for independent fields, and the silent pass a missing assertAll() produces.

for a principal

Weigh whether a team should limit soft asserts, since a test checking a dozen things softly blurs what it specifies and spreads one root cause across many reported failures.

## Two assertion styles in TestNG TestNG ships two ways to check a result inside a test method. **Hard asserts** are the static methods of `org.testng.Assert`: `assertEquals`, `assertTrue`, `assertNotNull`, `fail` and the rest. **Soft asserts** come from `org.testng.asserts.SoftAssert`, a class you instantiate and call methods on. Both use the same argument order (the actual value first, the expected value second, an optional message last) and the same failure text, `expected [x] but found [y]`. What differs is *when* a failed check ends the test. ## How a hard assert behaves A hard assert throws `java.lang.AssertionError` the moment its check fails. Nothing after that line runs, and TestNG records the test method as **failed** with that error's message and stack trace. - The first failure hides every later one: if three fields are wrong, the report shows only the first, and you discover the other two on later runs. - That is exactly what you want for a **precondition**. If a page did not load or a response body is null, every later check would fail for the same root cause and bury it in noise. - Hard asserts keep no state between calls, so they behave the same whether the suite runs sequentially or in parallel. ## How SoftAssert behaves A `SoftAssert` is a **collector**. Its lifecycle has three steps: 1. Create it inside the test method: `SoftAssert softly = new SoftAssert();`. 2. Call its checks: `softly.assertEquals(...)`, `softly.assertTrue(...)`, even `softly.fail(...)`. Each check runs; if it fails, the `AssertionError` is caught and stored in the order the checks ran, and the method carries on. 3. Call `softly.assertAll()` as the last statement. If anything was recorded, it throws a single `AssertionError` whose message starts with `The following asserts failed:` (or your own header, through `assertAll(String)`) and lists every recorded failure. The verdict is therefore **deferred**: the test method fails only when `assertAll()` runs, and the report shows every mismatch from that one run. ## What the failure looks like For a profile test where the name and the verified badge are both wrong, `assertAll()` throws an error like this: ```text java.lang.AssertionError: The following asserts failed: name expected [Ada Lovelace] but found [Ada], verified badge expected [true] but found [false] ``` Each line carries the message you passed to that check, which is why a short label per soft check pays off: without it, a list of `expected [true] but found [false]` lines is hard to map back to the code. ## Choosing between them | Situation | Better choice | Why | |---|---|---| | The object under test must exist before anything else is checked | Hard `Assert` | Later checks are meaningless without it | | One test verifies several independent fields of one result | `SoftAssert` | One run reports every mismatch | | A failed check would make the next line throw | Hard `Assert` | Avoids a cascade of follow-on errors | | A multi-step flow where each step depends on the previous one | Hard `Assert` | A soft failure would let the flow continue in a broken state | A common, sound pattern combines them: hard guards first, then a block of soft checks on independent properties, then `assertAll()` as the final line. ## The mistakes interviewers listen for - **Forgetting `assertAll()`.** TestNG does not call it for you. The failures are recorded and never thrown, the method returns normally, and the test **passes**. This is the most damaging soft-assert bug because nothing in the report hints at it. - **Expecting SoftAssert to catch exceptions.** It intercepts only the `AssertionError` raised by its own check methods. A `NullPointerException` from the code under test, or a hard `Assert` call, ends the method at once; `assertAll()` never runs, so the soft failures recorded so far are not reported. - **Making every check soft.** A test that continues past a broken precondition turns one root cause into a dozen reported failures and makes triage slower, not faster. - **Reusing one instance across test methods.** A `SoftAssert` never clears what it recorded, so an instance shared between methods carries one method's failures into the next. Create a fresh one in each test method. ## Contrast with JUnit Jupiter JUnit Jupiter has no collector object: it groups checks by passing lambdas to `Assertions.assertAll`, and its `assertEquals` takes the expected value first. Those mechanics belong to JUnit. In TestNG, the deferred verdict lives in the `SoftAssert` instance you create and in the `assertAll()` call you must not forget.

  • What happens if code between two soft checks throws a NullPointerException?
    The exception propagates immediately, exactly as it would without SoftAssert, because SoftAssert only catches the AssertionError raised by its own check methods. assertAll() is never reached, so soft failures recorded earlier are not reported; the test fails with only the NullPointerException. Put hard guards before the soft block so a missing object stops the test cleanly.
  • Does calling fail("...") on a SoftAssert end the test method?
    No. On a SoftAssert, fail() goes through the same record-and-continue path as every other check, so the failure is stored and the method keeps running; it surfaces only when assertAll() throws. The static Assert.fail() is the one that throws at once.
  • What does the message thrown by SoftAssert.assertAll() contain?
    By default it starts with 'The following asserts failed:' and then lists each recorded failure on its own line, in the order the checks ran, each with the label you passed to that check, such as 'email expected [[email protected]] but found [ada@test]'. assertAll(String) replaces the header with your own text.

A soft assert is like a building inspector filling in a defect checklist while walking every room: nothing is rejected until the checklist is handed in, and a checklist that is never handed in rejects nothing.

  • A SoftAssert failure fails the test as soon as that check runs.
  • TestNG calls assertAll() automatically when the test method returns.
  • SoftAssert also catches exceptions thrown by the code under test.
  • Soft asserts should replace hard asserts everywhere so each run shows more failures.
  • TestNG's Assert and SoftAssert take the expected value first, like JUnit Jupiter.
open as a page

In TestNG, in what order do the @BeforeSuite, @BeforeTest, @BeforeClass and @BeforeMethod hooks and their @After pairs run?

level: juniorimportance: must knowfreq 65%

basics

~10 s

TestNG nests its hooks from widest to narrowest: @BeforeSuite, @BeforeTest (once per <test> element of testng.xml), @BeforeClass, @BeforeMethod, then the test method; the @After hooks unwind in reverse, @AfterMethod first and @AfterSuite last.

open as a page

In TestNG, how does a @DataProvider method feed rows of arguments into a @Test method, and which return types may it use?

level: juniorimportance: must knowfreq 66%

basics

~10 s

A @DataProvider method returns rows of arguments, and every @Test method naming it in its dataProvider attribute runs once per row. Accepted return types are Object[][], Iterator<Object[]>, Object[] and Iterator<Object>.

open as a page

In TestNG, how do you tag test methods with groups and then run only one group, such as smoke, from testng.xml or the command line?

level: juniorimportance: must knowfreq 62%

basics

~20 s

Add groups = {"smoke"} to @Test, then select it with <groups><run><include name="smoke"/></run></groups> in testng.xml or -groups smoke on TestNG's command line. Group names match as regular expressions, and an excluded group beats an included one.

open as a page

In TestNG, which listener interfaces fire around a suite, a <test> element and a single test method, and how do you register one?

level: juniorimportance: must knowfreq 55%

basics

~10 s

TestNG's ISuiteListener wraps each suite, ITestListener wraps each <test> element and reports every test method's pass, failure or skip, and IInvokedMethodListener wraps every invocation. Register them with @Listeners, testng.xml's <listeners>, -listener or ServiceLoader.

open as a page

In a TestNG suite file (testng.xml), how do the suite, test, classes, methods and packages elements nest, and what do include and exclude select?

level: juniorimportance: must knowfreq 55%

basics

~10 s

A testng.xml holds one <suite> containing one or more <test> elements; each <test> lists <classes> (optionally narrowed by <methods>) or <packages>. Inside <methods>, include and exclude pick test methods by regular expression.

open as a page

Why might a Java team choose TestNG over JUnit Jupiter for a test suite, and when is JUnit Jupiter the better default?

level: juniorimportance: must knowfreq 45%

basics

~20 s

Teams pick TestNG for suite-level control: a testng.xml suite file, test-to-test dependencies, data providers, and suite and test-block hooks built in. JUnit Jupiter is the better default where the stack, frameworks and team already assume it.

open as a page

With TestNG's org.testng.Assert.assertEquals, in which order do the actual value, expected value and message go, and what breaks if they are swapped?

level: middleimportance: must knowfreq 50%

basics

~20 s

TestNG's Assert.assertEquals takes (actual, expected, message). Swapping the first two normally leaves pass or fail unchanged, but the failure text 'expected [x] but found [y]' then reports the values backwards and misleads whoever triages it.

open as a page

In TestNG, what does @Factory do, when would you choose it over a @DataProvider, and how can the two be combined?

level: middleimportance: must knowfreq 56%

basics

~20 s

@Factory creates test-class instances at run time and TestNG runs every @Test method on each. A @DataProvider repeats one method per row; a factory repeats a whole class, and @Factory(dataProvider = ...) on a constructor joins them.

open as a page

In TestNG, what does dependsOnMethods do, and what happens to dependent tests when the method they depend on fails?

level: middleimportance: must knowfreq 66%

basics

~20 s

dependsOnMethods makes a TestNG test run only after the named methods, and only if they all succeeded. If one fails, the dependent is reported SKIP, as is everything downstream, unless the dependent sets alwaysRun = true.

open as a page

How does TestNG's IRetryAnalyzer decide whether a failed test runs again, and how do the retried attempts show up in the results?

level: middleimportance: must knowfreq 52%

basics

~10 s

After a test attempt fails, TestNG calls its IRetryAnalyzer's retry(ITestResult); true re-runs the method. Each retried attempt is recorded as SKIP with wasRetried() true, so a fail-then-pass test shows one skip and one pass.

open as a page

In a TestNG suite file, what does each parallel value (methods, classes, tests, instances) keep on one thread, and what does thread-count size?

level: middleimportance: must knowfreq 62%

basics

~20 s

TestNG's parallel attribute picks the unit kept on one thread: methods keeps nothing together, classes keeps each class's methods, tests keeps each <test> element, instances keeps each object's methods. thread-count sizes the pool, default 5.

open as a page

In TestNG, how does @Parameters get values from testng.xml, which declaration wins on a name clash, and what does @Optional change?

level: middleimportance: must knowfreq 58%

basics

~20 s

@Parameters lists XML parameter names whose string values TestNG passes, in order, to the method's arguments. The innermost <parameter> wins (methods over class over test over suite); @Optional supplies a fallback when none is defined.

open as a page

In a TestNG Selenium-Java suite, the @BeforeClass login hook throws and all 40 tests in that class report SKIPPED instead of FAILED; why, and how do you diagnose it?

level: seniorimportance: must knowfreq 55%

basics

~20 s

TestNG treats a failed configuration method as a broken precondition: the hook is reported as a configuration failure and every test depending on it is SKIPPED. Diagnosis starts at the hook's exception, which each skipped result also carries.

open as a page

In TestNG, how does the `priority` attribute of `@Test` decide execution order, including for methods with no priority and for ties?

level: juniorimportance: should knowfreq 50%

basics

~10 s

TestNG schedules lower priority values first; the default is 0 and negative values are allowed. Ties fall back to name order, not source order, and dependencies and parallel runs can override the sequence.

open as a page

In TestNG's org.testng.Assert, what is the difference between assertThrows and expectThrows, and how do you check the thrown exception's message?

level: middleimportance: should knowfreq 38%

basics

~20 s

Both run a lambda and fail if it throws nothing or an exception that is not an instance of the expected class. assertThrows returns void; expectThrows returns the caught exception, so you can then assert on its message or cause.

open as a page

What does alwaysRun = true do on a TestNG configuration method, and why do teams put it on @AfterMethod and @AfterClass hooks?

level: middleimportance: should knowfreq 50%

basics

~20 s

TestNG documents alwaysRun = true two ways: an @After hook still runs after earlier methods failed or were skipped, and a @Before hook other than @BeforeGroups runs whatever its groups. Teams use it so teardown survives a skip cascade.

open as a page

In a TestNG @AfterMethod hook, how do you find out which test method just ran and whether it passed, failed or was skipped?

level: middleimportance: should knowfreq 42%

basics

~10 s

Declare an ITestResult parameter on the @AfterMethod method: TestNG injects the result of the test method that just ran, so getStatus(), getMethod() and getThrowable() give the outcome, the method and any exception.

open as a page

In TestNG, when a test class extends a base class and both declare @BeforeClass and @BeforeMethod hooks, in what order do they run?

level: middleimportance: should knowfreq 48%

basics

~10 s

Scope nests first, inheritance second: every class-level hook runs before any method-level one, and within one scope the superclass's @Before hook runs before the subclass's, while @After hooks run subclass first, then superclass.

open as a page

A TestNG @Factory builds one instance per browser and locale pair, but results show identical method names; how do you tell instances apart?

level: middleimportance: should knowfreq 38%

basics

~20 s

Implement org.testng.ITest so getTestName() returns, say, the browser and locale; TestNG uses it as each result's name. Without ITest, overriding toString() makes TestNG name results as the method name followed by "on" and that string.

open as a page

In TestNG, how do you reuse one @DataProvider across test classes, and how can it tailor its rows to the calling test method?

level: middleimportance: should knowfreq 46%

basics

~10 s

Put the provider in its own class and reference it with @Test(dataProviderClass = ..., dataProvider = ...). Declaring a java.lang.reflect.Method or ITestContext parameter on the provider lets it return different rows per calling test.

open as a page

In TestNG, what does an @BeforeGroups method do, when exactly does it run, and why might it never run in a group-filtered suite?

level: middleimportance: should knowfreq 42%

basics

~20 s

@BeforeGroups runs setup once per suite-file <test>, shortly before the first test method of the named groups. It never fires if no member of those groups is selected, or if its class is not part of that <test>.

open as a page

In TestNG, what can an IAnnotationTransformer and an IMethodInterceptor change about a run, and why is a transformer in @Listeners ignored?

level: middleimportance: should knowfreq 40%

basics

~10 s

IAnnotationTransformer rewrites annotation values, such as a @Test's retryAnalyzer or enabled flag, while TestNG parses classes; IMethodInterceptor reorders, filters or adds independent methods before they run. @Listeners is read too late for a transformer.

open as a page

A TestNG test uses `@Test(invocationCount = 50, threadPoolSize = 5)` to hammer an idempotent order endpoint; how does TestNG run it, and what must the test code guarantee?

level: middleimportance: should knowfreq 42%

basics

~20 s

TestNG invokes the method 50 times on a pool of 5 threads, all on the same test-class instance, recording each invocation as its own result. The code must be thread-safe: no unsynchronised shared instance state, and assertions about each call.

open as a page

In TestNG's `@Test`, how do `timeOut` and `invocationTimeOut` differ, and what happens to a test that exceeds either limit?

level: middleimportance: should knowfreq 40%

basics

~20 s

timeOut caps one invocation and invocationTimeOut caps the combined time of all invocationCount runs, both in milliseconds. Breaching either marks the test FAILED with a ThreadTimeoutException; TestNG interrupts the method's thread but cannot kill it.

open as a page

When mapping a TestNG suite onto JUnit Jupiter, what replaces groups, @DataProvider, @Factory and testng.xml, and what has no equivalent?

level: middleimportance: should knowfreq 40%

basics

~10 s

Groups become @Tag, @DataProvider becomes @ParameterizedTest with @MethodSource, @Factory is closest to the experimental @ParameterizedClass, and testng.xml becomes a @Suite class plus configuration parameters. dependsOnMethods and @BeforeTest have no direct equivalent.

open as a page

A TestNG class holds one SoftAssert as a field, and later test methods fail with messages from checks they never made; why, and how do you fix it?

level: seniorimportance: should knowfreq 40%

basics

~20 s

TestNG runs all of a class's test methods on one instance by default, so a SoftAssert field is shared, and assertAll() never clears what it recorded. Later methods rethrow earlier failures. Create a new SoftAssert inside each test method.

open as a page

A TestNG login matrix reads a few hundred rows from an Iterator-returning @DataProvider(parallel = true); results turn flaky while threads and memory climb. What is going on?

level: seniorimportance: should knowfreq 44%

basics

~20 s

With parallel = true, rows run concurrently on pool threads against one shared test instance, so instance fields race. Each such method gets its own pool unless pooling is shared, and the iterator is drained into a queue.

open as a page

After a TestNG suite starts excluding a group, the run errors with 'depends on nonexistent group'; why does dependsOnGroups do that, and how would you fix it, including with ignoreMissingDependencies?

level: seniorimportance: should knowfreq 36%

basics

~20 s

dependsOnGroups resolves against the methods selected for the run, so excluding the depended-on group leaves nothing to match and TestNG throws a TestNGException. Restore the selection, or set ignoreMissingDependencies = true on the dependent to run without it.

open as a page

A flaky TestNG checkout UI suite applies IRetryAnalyzer to every test through an IAnnotationTransformer, and a listener screenshots in onTestFailure: what goes wrong?

level: seniorimportance: should knowfreq 42%

basics

~20 s

TestNG ignores a transformer declared in @Listeners, so retries may never happen. Once they do, retried attempts become SKIP and fire onTestSkipped, so onTestFailure screenshots miss every flaky attempt and a fail-then-pass test looks green.

open as a page

showing 1–30 of 40