skip to content

In TestNG, which listener interfaces fire around a suite, a <test> element and a single test method, and how do you register one?

level: juniorimportance: must knowfreq 55%

answer

  1. observe the run at fixed points
  2. three scopes, three interfaces
  3. result callbacks per test method
  4. annotation, suite file, classpath

basics

~10 s

TestNG's ISuiteListener wraps each suite, ITestListener wraps each <test> element and reports every test method's pass, failure or skip, and IInvokedMethodListener wraps every invocation. Register them with @Listeners, testng.xml's <listeners>, -listener or ServiceLoader.

solid answer

~30 s

A TestNG listener implements a sub-interface of `org.testng.ITestNGListener`, and TestNG calls it at fixed points. `ISuiteListener.onStart/onFinish(ISuite)` wraps a suite. `ITestListener.onStart/onFinish(ITestContext)` wraps one `<test>` element, and its `onTestStart`, `onTestSuccess`, `onTestFailure` and `onTestSkipped` fire per test-method result, which is where a screenshot-on-failure hook usually sits. `IInvokedMethodListener.beforeInvocation/afterInvocation` wraps every invocation, configuration methods included, `IConfigurationListener` covers only the `@Before*`/`@After*` hooks, and `IReporter.generateReport` runs once after all suites. Register a listener with `@Listeners` on a class (it applies suite-wide), a `<listener class-name=...>` in `testng.xml`, the `-listener` option, or a `META-INF/services/org.testng.ITestNGListener` file.

code

java · 16 lines
java
import org.testng.ITestContext;
import org.testng.ITestListener;
import org.testng.ITestResult;

public class FailureLogListener implements ITestListener {
  @Override
  public void onStart(ITestContext context) {
    System.out.println("<test> started: " + context.getName());
  }

  @Override
  public void onTestFailure(ITestResult result) {
    System.out.println("FAILED " + result.getMethod().getMethodName()
        + ": " + result.getThrowable());
  }
}

go deeper

for a junior

Recall the three scopes and their interfaces: suite, <test> element, test method. Know that onTestFailure is where a failure screenshot usually goes and that @Listeners or testng.xml registers it.

for a middle

Explain that ITestListener.onStart wraps a <test> element, that IInvokedMethodListener also sees configuration methods, and that @Listeners applies to the whole suite rather than one class.

for a senior

Show you can diagnose a listener firing twice or not at all: class-keyed de-duplication, two classes doing one job, a listener needing to filter isTestMethod(), a transformer ignored in @Listeners.

for a principal

Talk about distributing shared listeners across teams: a ServiceLoader jar versus per-suite declarations, ordering them with a ListenerComparator, and keeping callbacks cheap because they run for every method.

## What a TestNG listener is A **listener** in TestNG is a class that implements one of the sub-interfaces of `org.testng.ITestNGListener`. TestNG calls its methods at fixed points of a run. Most listeners **observe**: they log, take a screenshot, collect timings or write a report. A few listener types **change** the run instead (an `IAnnotationTransformer` rewrites annotation values, an `IMethodInterceptor` reorders the method list), but the everyday ones only watch. Almost every callback on the common interfaces has a default implementation, usually empty, so an implementation overrides only what it needs. ## Which interface fires when TestNG runs a **suite** (one `<suite>` in `testng.xml`), which contains one or more **`<test>` elements**, which contain classes and their **test methods**. Each scope has its own interface: | Interface | Callbacks | Fires | |---|---|---| | `IExecutionListener` | `onExecutionStart`, `onExecutionFinish` | once around the whole TestNG run | | `ISuiteListener` | `onStart(ISuite)`, `onFinish(ISuite)` | around each suite | | `ITestListener` | `onStart(ITestContext)`, `onFinish(ITestContext)` | around each `<test>` element | | `ITestListener` | `onTestStart`, `onTestSuccess`, `onTestFailure`, `onTestSkipped` | for each test-method result | | `IInvokedMethodListener` | `beforeInvocation`, `afterInvocation` | around every invocation, configuration methods included | | `IConfigurationListener` | `beforeConfiguration`, `onConfigurationSuccess`, `onConfigurationFailure`, `onConfigurationSkip` | around `@Before*` / `@After*` methods only | | `IReporter` | `generateReport(xmlSuites, suites, outputDirectory)` | once, after all suites have run | Points that interviewers probe: - **`ITestListener` is the result listener.** `onTestFailure(ITestResult)` receives the result with its throwable, which is why a screenshot-on-failure hook usually lives there. - **Its `onStart(ITestContext)` is not per method.** It runs before all the methods of one `<test>` element, which surprises people who expect "start" to mean "before each test". - **`IInvokedMethodListener` sees configuration methods too.** Use `IInvokedMethod.isTestMethod()` or `isConfigurationMethod()` to filter. By default it also fires for skipped methods; the `-alwaysrunlisteners` option turns that off. - **`ITestListener` also has `onTestFailedButWithinSuccessPercentage` and `onTestFailedWithTimeout`**; the latter forwards to `onTestFailure` unless overridden. - **`IReporter` is post-mortem.** It gets the full results of every suite and an output directory, and nothing runs after it. ## What a listener receives - **`ITestResult`** carries the method (`getMethod()`), its status (`SUCCESS`, `FAILURE`, `SKIP`), the throwable (`getThrowable()`), the parameters and the timing, and lets a listener store values with `setAttribute`. - **`ITestContext`** describes one `<test>` element: its name, its passed, failed and skipped results (`getPassedTests()`, `getFailedTests()`, `getSkippedTests()`) and shared attributes. - **`ISuite`** describes the suite and, once it has run, the results of each `<test>` element in it. - **`IInvokedMethod`** wraps the invoked method and says whether it was a test or a configuration method. The per-method callbacks run on the worker thread that ran the method, so a listener that keeps its own state across methods must make that state thread-safe when the suite runs methods in parallel. ## Registering a listener There are four common paths: 1. **`@Listeners({MyListener.class})`** on a test class. Despite sitting on one class, the documentation says it applies to the whole suite, as if declared in `testng.xml`. 2. **`<listeners><listener class-name="com.example.MyListener"/></listeners>`** inside `<suite>` in `testng.xml`. 3. **The `-listener` option** of the `org.testng.TestNG` command-line runner, taking comma-separated class names. 4. **ServiceLoader**: a jar on the classpath carrying `META-INF/services/org.testng.ITestNGListener` with the class name in it. This is how one listener reaches every team's suite without anyone editing their files. The exception is `IAnnotationTransformer`: TestNG must know it before it parses annotations, so the documentation says a transformer placed in `@Listeners` is ignored. Register it in `testng.xml`, on the command line or in code. ## Scope and duplicate registration - TestNG keys the common listener types by their **class**, so the same class registered through `@Listeners` and `<listeners>` ends up as one instance; a duplicate caught at TestNG's top-level registration logs `Ignoring duplicate listener`. - Two **different** classes doing the same job are two keys, and both run. A base listener and a team copy of it will both take a screenshot. - One class implementing two interfaces that both act on a failure, say `onTestFailure` and `afterInvocation`, also captures twice. - Because `@Listeners` is suite-wide, a listener meant for one class must check the method it receives and return early for others. - Since TestNG 7.10.0 a `ListenerComparator`, passed with `-listenercomparator`, can fix the order of your own listeners. ## Common mistakes - Expecting `ITestListener.onStart` to run before each test method. - Putting a heavy per-method action in `IInvokedMethodListener` and running it for every configuration method as well. - Believing `@Listeners` limits a listener to the annotated class. - Importing a JUnit Jupiter extension interface and expecting TestNG to call it: TestNG calls only its own `org.testng` listener types.

  • What happens if the same TestNG listener class is registered both with @Listeners and in testng.xml?
    TestNG keys the common listener types by class, so the second registration of the same class is dropped and one instance runs; a duplicate caught at TestNG's top-level registration logs `Ignoring duplicate listener`. Two different classes that do the same work are separate keys, so both fire, which is how duplicate screenshots usually appear.
  • How would you roll one TestNG listener out to many teams' suites without editing each testng.xml?
    Package it in a jar with a `META-INF/services/org.testng.ITestNGListener` file naming the class, and put the jar on the test classpath. TestNG discovers ServiceLoader listeners at start-up. The `-spilistenerstoskip` option lets a run exclude a discovered listener by class name.

  • ITestListener.onStart runs before every test method
  • @Listeners on a class limits the listener to that class
  • IInvokedMethodListener fires only for @Test methods
  • IReporter is called after each test method
  • Registering a listener twice always makes it fire twice

Assumes TestNG 7.12; the listener interfaces are long-standing, and ListenerComparator ordering exists since 7.10.0.