skip to content

A TestNG validation test with `@Test(expectedExceptions = IllegalArgumentException.class)` stayed green after the validation it covers was deleted; how can that happen, and how do you harden it?

level: seniorimportance: should knowfreq 40%

answer

  1. any line in the method
  2. subclasses also match
  3. a parse error is an IllegalArgumentException
  4. the message regex must match fully

basics

~20 s

expectedExceptions passes when any statement in the method throws the declared type or a subclass, so a NumberFormatException from parsing test data satisfies IllegalArgumentException. Harden it with a full-match message regex and by asserting on the one call that must throw.

solid answer

~30 s

TestNG checks only that the method ended by throwing something assignable to a type in `expectedExceptions`. It does not know which line threw, and it accepts subclasses, so a `NumberFormatException` thrown while parsing a fixture value, before the service is even called, satisfies `IllegalArgumentException`. To harden it: add `expectedExceptionsMessageRegExp`, which must match the **whole** message (and fails on a null message); keep the method body to the single call under test with all setup outside it; or assert the exception around exactly that call with TestNG's `assertThrows` or `expectThrows`. Then prove the test by deleting the validation and watching it fail.

code

java · 13 lines
java
import org.testng.annotations.Test;

public class OrderValidationWrongReasonTest {

  private final OrderService orders = new OrderService();

  @Test(expectedExceptions = IllegalArgumentException.class)
  public void rejectsNegativeQuantity() {
    String raw = "-2.5";                    // fixture typo
    int quantity = Integer.parseInt(raw);   // NumberFormatException
    orders.place("SKU-1", quantity);        // never reached
  }
}

go deeper

for a junior

Know that expectedExceptions makes a test pass when the method throws the listed type, and fail when it throws nothing.

for a middle

Explain subclass matching and the full-match, DOTALL message regex, including the null-message case.

for a senior

Diagnose a test that passes for the wrong reason, shrink the throwing region, and verify it by deleting the validation and watching it fail.

for a principal

Decide whether the team still allows the attribute for new tests or prefers assertions around the single call, and how review catches broad declared types.

## What `expectedExceptions` actually checks `expectedExceptions` is a `Class[]` attribute on TestNG's `@Test`: *"The list of exceptions that a test method is expected to throw. If no exception or a different than one on this list is thrown, this test will be marked a failure."* TestNG invokes the method, catches whatever escapes it, and asks one question: is the thrown exception's class **assignable to** any class in the list? In TestNG's source this is an `isAssignableFrom` check, so subclasses match. Three outcomes are possible: | What the method does | Status TestNG records | |---|---| | Throws a listed type or a subclass of one | Success | | Throws an unrelated type | Failure: expected one type but got another (or the wrong-message failure when a regex is set) | | Completes without throwing | Failure: the method should have thrown | None of these involves *where* the exception came from. That is the whole problem. ## How a deleted validation stays green Consider an order-service test whose fixture value was meant to be `"-2"` but was typed as `"-2.5"`: ```java @Test(expectedExceptions = IllegalArgumentException.class) public void rejectsNegativeQuantity() { String raw = "-2.5"; // fixture typo int quantity = Integer.parseInt(raw); // throws NumberFormatException orders.place("SKU-1", quantity); // never reached } ``` `Integer.parseInt` throws `NumberFormatException`, which **extends** `IllegalArgumentException`. The test passes on line two. The service is never called, so deleting its validation changes nothing. Common variants: - a test-data builder that rejects an argument with its own `IllegalArgumentException`; - a `NullPointerException` from a missing mock when the expected type is the broad `RuntimeException`; - an `expectedExceptions = Exception.class` that accepts almost anything. A related trap: any assertion written **after** the throwing call is dead code, because the method exits at the throw. Engineers sometimes add `Assert.assertEquals(repository.count(), 0)` under the call and believe they verified that nothing was saved. ## Hardening, in order of strength 1. **Narrow the declared type.** Declare the most specific exception the code under test throws, not `Exception` or `RuntimeException`. 2. **Pin the message.** `expectedExceptionsMessageRegExp` (default `".*"`, meaning "do not check") must match the thrown message. TestNG compiles it with `Pattern.DOTALL` and calls `matches()`, so the regex has to match the **entire** message, not a fragment: `"quantity"` fails against `"quantity must be positive"`, while `"quantity must be positive.*"` passes. If the regex is not the default and the exception's message is `null`, the check fails. A mismatch is reported as a failure saying the exception was thrown with the wrong message. 3. **Shrink the region that may throw.** Move parsing and building into fields, setup hooks or a data source, so the method body is one call. 4. **Assert around the one call.** TestNG's `Assert.assertThrows` and `Assert.expectThrows` wrap a lambda containing only the call under test and let you inspect the returned exception; how those asserts work belongs with TestNG's assertion API. ## Reading the failure messages When the attribute does catch a problem, TestNG's message tells you which of three things went wrong, and each points to a different fix: - *"Method ... should have thrown an exception of type ..."*: nothing was thrown, so the validation is missing or the input did not trigger it. - *"Expected exception of type ... but got ..."*: something else was thrown, often a setup error such as a null client or a missing stub. - *"The exception was thrown with the wrong message"*: the message did not match the regex. TestNG uses this text whenever the regex fails, even if the type was wrong too, so read the actual exception it prints next to the regex. The dangerous case is the one with no message at all, a green run, which is why the checks below matter more than the attribute itself. ## Proving the test The decisive habit is a quick mutation check: comment out the validation, run the test, and watch it fail. A test that stays green under that change is not testing the validation. ## Example of the tightened version ```java import org.testng.annotations.Test; public class OrderValidationTest { private final OrderService orders = new OrderService(); @Test( expectedExceptions = IllegalArgumentException.class, expectedExceptionsMessageRegExp = "quantity must be positive.*") public void rejectsNegativeQuantity() { orders.place("SKU-1", -2); } } ``` The body is one call, the literal needs no parsing, and the message must start with the validation's wording and match in full. ## The JUnit contrast in one line JUnit 4 had `@Test(expected = ...)` with the same weakness, and JUnit Jupiter dropped the attribute in favour of an assertion around the call; TestNG keeps the attribute, so the discipline above is the TestNG user's job.

  • Does TestNG's expectedExceptionsMessageRegExp match a fragment of the exception message?
    No. TestNG compiles it with `Pattern.DOTALL` and calls `matches()`, so it must match the entire message; wrap a fragment as `.*fragment.*`. The default `.*` skips the check, and with any other regex a `null` message fails.
  • How does TestNG report a test with expectedExceptions that completes without throwing?
    As a failure. TestNG raises a `TestException` stating that the method should have thrown an exception of the declared type, so a missing exception can never pass silently.
  • Why do assertions written after the expected-throwing call in a TestNG test never verify anything?
    The method exits at the throw, so later statements never execute; TestNG only sees the exception and passes the test. To check state after the failure, assert the exception around the call with `expectThrows` and verify state afterwards in the same method.

  • expectedExceptions only matches the exact declared class, not its subclasses.
  • expectedExceptionsMessageRegExp passes if the regex matches part of the message.
  • A test with expectedExceptions that throws nothing is marked SKIPPED.
  • Assertions written after the throwing call still run and verify state.
  • expectedExceptions checks that the exception came from the method under test.