skip to content

A TestNG test uses `@Test(invocationCount = 50, threadPoolSize = 5)` to hammer an idempotent order endpoint; how does TestNG run it, and what must the test code guarantee?

level: middleimportance: should knowfreq 42%

answer

  1. count versus pool size
  2. one shared test instance
  3. each run gets its own result
  4. a percentage of allowed failures

basics

~20 s

TestNG invokes the method 50 times on a pool of 5 threads, all on the same test-class instance, recording each invocation as its own result. The code must be thread-safe: no unsynchronised shared instance state, and assertions about each call.

solid answer

~40 s

`invocationCount` is how many times the method runs; `threadPoolSize` is how many threads share those runs. With 50 and 5, TestNG creates one worker per invocation, runs them on a pool of five threads that start together like a small load burst, and records 50 separate results. Every invocation is a clone of the same method on the **same test-class instance**, so an instance field written by one thread is visible to and raced by the others; keep per-call state in local variables and use thread-safe collectors for anything aggregated. The Javadoc says `threadPoolSize` is ignored when `invocationCount` is not specified. For an idempotency check, assert on each response and then verify one order exists. `successPercentage` lowers the bar from 100, recording tolerated failures under their own status, not as passes.

code

java · 16 lines
java
import java.util.Set;
import java.util.concurrent.ConcurrentHashMap;
import org.testng.Assert;
import org.testng.annotations.Test;

public class OrderIdempotencyTest {

  private final Set<String> orderIds = ConcurrentHashMap.newKeySet();

  @Test(invocationCount = 50, threadPoolSize = 5)
  public void sameKeyReturnsAnOrder() {
    String orderId = OrderStub.place("idem-key-7", "SKU-1", 2);
    Assert.assertNotNull(orderId);
    orderIds.add(orderId);
  }
}

go deeper

for a junior

Recall that invocationCount is the number of runs and threadPoolSize the number of threads sharing them, and that the total is not multiplied.

for a middle

Explain the pooled path: clones of one method on one shared instance, one result per invocation, and why threadPoolSize needs invocationCount.

for a senior

Diagnose flaky pooled tests caused by instance fields and non-thread-safe collections, and read successPercentage and skipFailedInvocations correctly in a report.

for a principal

Set the boundary: pooled invocations are a concurrency smoke for idempotency and races, kept small, while capacity questions go to a dedicated load tool.

## The two attributes - **`invocationCount`** (default `1`): *"The number of times this method should be invoked."* - **`threadPoolSize`** (default `0`): *"The size of the thread pool for this method. The method will be invoked from multiple threads as specified by invocationCount."* Its Javadoc adds that it *"is ignored if invocationCount is not specified"*. Together they let a single `@Test` method fire many concurrent calls. In TestNG's invoker, when `invocationCount > 1` and `threadPoolSize > 1`, the method takes a pooled path: TestNG clones the method once per invocation, gives each clone an invocation count of one, wraps each in a worker, and runs the workers on a pool of `threadPoolSize` threads. An old changelog entry describes these "load @Tests" as triggered simultaneously, and the pool's startup is synchronised so that the calls begin together. | Attributes | What runs | |---|---| | `invocationCount = 50` | 50 sequential invocations on one thread | | `invocationCount = 50, threadPoolSize = 5` | 50 invocations spread over 5 threads | | `threadPoolSize = 5` alone | One invocation; the pool size is ignored | The total is always `invocationCount`. It is not multiplied by the pool size. ## The shared-instance trap Each clone keeps the **same test-class instance**. TestNG does not construct a new object per invocation, so: - an instance field such as `private Response last;` is overwritten by five threads in turn, and an assertion can read another thread's response; - a plain `ArrayList` used to collect order IDs can lose entries or throw under concurrent `add`; - any client object stored in a field must itself be safe for concurrent use. The fixes are ordinary concurrency hygiene: keep each call's request and response in local variables, collect into `ConcurrentHashMap` or `ConcurrentLinkedQueue`, and give each thread its own non-thread-safe client if needed. ## Designing the idempotency check For the reserved scenario, an endpoint that must create exactly one order however many times the same idempotency key is sent: 1. Each invocation sends the same request with the same key and asserts on its own response, for example that the status is success and the returned order ID is present. 2. Each invocation adds the returned order ID to a thread-safe set held by the class. 3. A separate check, run after the 50 invocations, asserts that the set holds exactly one ID. Step 3 needs an ordering mechanism to run after the pooled method, which is a separate subject; the point here is that the pooled method itself must only assert what is true of one call. ## Reading the results Every invocation produces its own result, so a run with three failing calls shows three failures and 47 passes for the one method. Two attributes change how failures are recorded: - **`successPercentage`** (default `100`): *"The percentage of success expected from this method."* For a sequential method, TestNG allows `(100 - successPercentage) * invocationCount / 100` failures. With `invocationCount = 10, successPercentage = 80` the allowance is two, so the first two failing invocations get the status `SUCCESS_PERCENTAGE_FAILURE` ("failed but within success percentage"), and the third and later failures are plain `FAILURE`. Tolerated failures are not turned into passes: listeners receive `onTestFailedButWithinSuccessPercentage` and reports count them separately. - **`skipFailedInvocations`** (default `false`): per its Javadoc, when `true` and `invocationCount` is above 1, all invocations after a failure are marked SKIP instead of FAIL, which saves time when one failure already proves the point. ## What this is not Five threads inside a test JVM are a concurrency smoke test, not a performance test. They can expose a missing idempotency guard or a race in the endpoint, but they say nothing reliable about throughput or latency percentiles. Keep the count modest so the suite stays fast, and use a dedicated load tool for capacity questions. A pooled method that fails intermittently is also a signal worth reading carefully: it may be a real race in the endpoint, or simply a race in the test's own shared fields, and the second must be ruled out before the first is reported. ## Example ```java import java.util.Set; import java.util.concurrent.ConcurrentHashMap; import org.testng.Assert; import org.testng.annotations.Test; public class OrderIdempotencyTest { private final Set<String> orderIds = ConcurrentHashMap.newKeySet(); @Test(invocationCount = 50, threadPoolSize = 5) public void sameKeyReturnsAnOrder() { String orderId = OrderStub.place("idem-key-7", "SKU-1", 2); Assert.assertNotNull(orderId); orderIds.add(orderId); } } ``` `orderId` is local to each invocation; only the concurrent set is shared, and it is safe for that.

  • On a sequential TestNG @Test(invocationCount = 10, successPercentage = 80), how many failures are tolerated?
    Two. TestNG allows `(100 - 80) * 10 / 100 = 2` failures, so the first two failing invocations are recorded as failed but within success percentage, and any further failure is a plain failure. Tolerated failures are still visible as their own status, not as passes.
  • What happens if a TestNG method sets threadPoolSize = 5 but no invocationCount?
    The Javadoc says `threadPoolSize` is ignored when `invocationCount` is not specified, and the invoker takes the pooled path only when both are above one. The method runs once, on whatever thread the suite would use anyway.
  • What does skipFailedInvocations = true change for a repeated TestNG method?
    Per its Javadoc, when `invocationCount` is above one, every invocation after a failure is marked SKIP instead of FAIL. It shortens a run that one failure has already decided and stops a single defect from producing dozens of identical failures.

  • Each pooled invocation gets a fresh instance of the test class.
  • threadPoolSize on its own makes one test run on several threads.
  • invocationCount = 50 with threadPoolSize = 5 means 250 calls in total.
  • A successPercentage below 100 turns tolerated failures into passes.
  • Fifty pooled invocations give a meaningful throughput benchmark.