A TestNG login matrix reads a few hundred rows from an Iterator-returning @DataProvider(parallel = true); results turn flaky while threads and memory climb. What is going on?
answer
- rows share one test instance
- separate pool per provider method
- data-provider-thread-count defaults to 10
- iterator drained into a queue
- shared pool attributes since 7.9.0
basics
~20 sWith parallel = true, rows run concurrently on pool threads against one shared test instance, so instance fields race. Each such method gets its own pool unless pooling is shared, and the iterator is drained into a queue.
solid answer
~40 sThree things are at work. First, every row invokes the test on the **same test-class instance**, so any field such as a driver, a page object or a current user is shared by concurrent rows; that is the flakiness, and per-row state belongs in locals. Second, each method with a parallel provider runs its rows on a fixed pool sized by the suite attribute `data-provider-thread-count` (default 10), and unless `share-thread-pool-for-data-providers` or `use-global-thread-pool` is set (both since 7.9.0), each such method creates its own pool, so with suite-level parallel methods the thread count multiplies. Third, in parallel mode TestNG walks the whole iterator on the calling thread and queues one task per row before waiting, so all rows sit in memory at once.
code
java · 32 linesimport java.util.Iterator;
import java.util.List;
import org.testng.annotations.DataProvider;
import org.testng.annotations.Test;
public class LoginMatrixTest {
private String currentUser; // shared by every concurrent row: a race
@DataProvider(parallel = true)
public Iterator<Object[]> credentials() {
List<Object[]> rows = List.of(
new Object[] {"alice", "pw1"},
new Object[] {"bob", "pw2"});
return rows.iterator();
}
@Test(dataProvider = "credentials")
public void racyLogin(String user, String password) {
currentUser = user; // another row may overwrite this
submit(currentUser, password);
}
@Test(dataProvider = "credentials")
public void safeLogin(String user, String password) {
String rowUser = user; // local: private to this row
submit(rowUser, password);
}
private void submit(String user, String password) {
}
}go deeper
Recall that parallel = true on a @DataProvider runs its rows concurrently, sized by the suite attribute data-provider-thread-count, ten by default.
Explain that rows share one test instance, that the provider's pool is separate from thread-count by default, and that an iterator loses its laziness when rows run in parallel.
Diagnose the trio: instance-field races, per-method pools multiplying threads, and queued rows inflating memory; fix with locals and a shared, sized pool.
Weigh provider-level parallelism against suite-level parallelism for the team: speed against predictable load on browsers, grids and the machines running them.
## The scenario A UI suite checks a login form against a few hundred credential rows. The rows are read lazily from a file through a provider returning `Iterator<Object[]>`, and someone adds `parallel = true` to the `@DataProvider` to cut the run time. The run gets faster, but some rows now fail at random with another row's user in the browser, the machine shows far more threads than expected, and memory rises during the run. Each symptom has a specific cause in how TestNG executes a parallel data provider. ## Cause 1: every row shares one test instance A data provider multiplies **invocations**, not instances. All rows of a test method are invoked on the **same test-class object**, sequentially or in parallel. With `parallel = true`, those invocations run concurrently on pool threads, so: - a field holding a browser session, a page object or "the current user" is written by one row while another row reads it; - `@BeforeMethod` and `@AfterMethod` run around each row on that row's worker thread, so anything they store in a field is shared in the same way; - counters and collections on the instance need to be thread-safe or must go. The TestNG-level fix is to keep per-row state **local to the test method** (create it from the row's arguments inside the method), or to hold it in a per-thread container if a hook must create it. How a whole harness isolates state across threads is a broader design question; the TestNG fact to state is that rows share the instance. ## Cause 2: one pool per provider method By default, a parallel provider does not use the suite's `thread-count`. Its rows run on a **fixed thread pool** whose size is the suite attribute `data-provider-thread-count`, **10 by default**, overridable by the `dataproviderthreadcount` system property. In the TestNG 7.12 source, the pool is created per invocation of a method with a parallel provider, unless pooling is shared: | Suite attribute (since 7.9.0) | Effect | |---|---| | neither set (default) | each method with a parallel provider gets its own pool of `data-provider-thread-count` threads | | `share-thread-pool-for-data-providers="true"` | all data-driven tests in the suite reuse one pool sized by `data-provider-thread-count` | | `use-global-thread-pool="true"` | regular and data-driven tests share one pool sized by `thread-count` | So if the suite also runs methods in parallel with `thread-count="5"` and five data-driven methods start together, the default can mean the five suite threads plus five pools of ten. That is where the thread count climbs, and with browsers behind each thread it is also where the grid or machine runs out of capacity. ## Cause 3: the iterator is no longer lazy In a **sequential** run TestNG pulls one row from the iterator per invocation, which is why an iterator provider can stream a large file. In a **parallel** run, TestNG's runner loops over the iterator on the calling thread and submits one task per row to the pool without waiting, then waits for all results. Because the pool's queue is unbounded, the whole file is effectively read up front and every row's arguments sit in queued tasks until a thread picks them up. Two corollaries: 1. The iterator itself is consumed by a single thread, so it does not need to be thread-safe; the test body does. 2. If rows are large (screenshots, payloads), memory grows with the row count, not with the thread count. ## A diagnosis checklist - Search the test class and its hooks for **instance fields** written per row. - Check the suite file for `data-provider-thread-count`, `thread-count`, `parallel` and the two sharing attributes, and compute the worst-case thread total. - Measure memory against **row count** to confirm queued rows rather than a leak. - Consider whether `parallel = true` is needed on this provider, or whether a smaller shared pool gives most of the speed with predictable load. - Rerun the method with `parallel = true` removed: if the flakiness disappears while the rows stay the same, shared state is confirmed as the cause rather than the application under test. ## What the report shows Parallel rows still produce one result per row with its parameters. A row that fails because another row overwrote shared state is reported as a **failure** of that row, which is why the defect reads as flakiness rather than as a configuration error.
- Does a TestNG Iterator provider marked parallel = true need a thread-safe iterator?Not for the iteration itself: in the 7.12 source the calling thread walks the iterator and submits one task per row, so only one thread calls `hasNext()` and `next()`. What runs concurrently is the test method, its per-row hooks and anything they share on the test instance.
- Does setting the suite's thread-count change how many threads a parallel TestNG data provider uses?Not by default. A parallel provider's rows run on a pool sized by `data-provider-thread-count` (default 10). Only with `use-global-thread-pool="true"`, available since 7.9.0, do regular and data-driven tests share a single pool sized by `thread-count`.
- Why can a parallel TestNG provider exhaust a browser grid even with a small thread-count?Because, without the sharing attributes, every method with a parallel provider gets its own pool of `data-provider-thread-count` threads, and those pools add to the suite's own threads. Several data-driven methods starting together can each open ten browser sessions.
- Each data provider row gets its own fresh instance of the test class
- By default a parallel data provider's pool is sized by the suite's thread-count
- An Iterator provider stays lazy when parallel = true is set
- parallel = true on a provider only works when the suite file sets parallel
- The iterator must be synchronized because pool threads call next() concurrently
Assumes TestNG 7.12; the share-thread-pool-for-data-providers and use-global-thread-pool suite attributes exist since 7.9.0.