How do you shard a long-running automated suite across parallel workers so that wall-clock time actually drops?

## Start from the budget, not the shard count A suite has a feedback budget: the time a team is willing to wait before the result changes what they do next. A change-gating suite might get ten minutes; a nightly pack might get a night. Sharding is one of several instruments for fitting a suite into its budget, and it is the one people reach for first because it requires no decisions about the cases themselves. That is exactly why it is worth being precise about when it works. ## Wall clock is the longest shard The number that matters is the critical path, not the total. Split 3,182 cases eight ways by file name and the shards will not be equal, because case durations are not uniformly distributed: a handful of long journeys sit somewhere in the alphabet. One real suite over a clinical-trial data capture form split this way produced shards ranging from 41 minutes to 96 minutes. Total processor time was unchanged and the dashboard reported a healthy eight-way split, but the run still took 96 minutes, because seven workers sat idle waiting for the eighth. The perfectly balanced split would have been about 47 minutes each. Two balancing strategies fix that. **Static balancing by measured duration.** Record each case's duration on every run, then assign cases to shards greedily, longest first, always into the shard with the smallest running total. This is the classic longest-processing-time heuristic and it gets close to optimal for this shape of problem. It needs a durable store of durations keyed by a stable case identifier - which is why renaming cases wholesale quietly degrades balance until the next run's data arrives. **Dynamic balancing by pull.** Put every case on a shared queue and let each worker take the next one when it becomes free. This self-corrects without historical data and copes with machines of different speeds, at the cost of a coordinator and of losing the ability to know in advance which case runs where. A common hybrid pulls in duration order, longest first, so the long tail starts early. ## The floor no sharding beats If one case takes 38 minutes, no shard count gets the run under 38 minutes. Before adding workers, look at the distribution: the top 2 percent of cases by duration often own a quarter of the run. That time usually goes into setup that could be shared, waiting that could be signalled rather than slept through, or a journey that is really four independent checks glued together. Splitting or relocating a handful of cases can beat doubling the worker count, and it costs nothing per run afterwards. ## Independence is the hard half Shards are only parallel if they cannot interfere. The recurring collisions are mundane: - **Shared mutable records.** Two shards creating a subject record with the same identifier will fail in a pattern that neither shard reproduces alone. Give each shard its own namespace - an identifier prefix, a schema, a tenant - or generate identifiers that cannot collide. - **Fixed resources.** A hard-coded port, a single service account with a session limit, one seeded login used by every case, a single output directory. - **Ordering assumptions.** A case that only passes because another case ran first was already fragile; sharding is simply the moment it is discovered, since the two land on different workers. - **Global state in the system under test.** A feature flag toggled by one case, a locale or clock setting changed globally, a cache cleared mid-run. A useful preparation step before sharding at all is to shuffle case order within a single worker: anything that fails under shuffling will fail under sharding too, and diagnosing it in one process is far easier. ## Reporting and cost Each shard produces its own results and artefacts, so publish them into one aggregated report keyed by case, not per worker - otherwise triage begins with finding which of eight reports holds the failure. Name artefacts with the shard and run so they cannot overwrite one another. If shards retry internally, keep the retry visible in the aggregate: a run that is green only because one shard passed on its third attempt is not the same result as a clean run, and hiding that turns a maintenance signal into noise. Finally, shards cost money and environments. Eight workers usually means eight application instances or eight data namespaces, and the marginal minute saved gets more expensive as the split gets finer. Past a point the honest answer is not more shards but fewer cases: prune duplication, move checks to a cheaper level, and reserve the wide split for the tier that genuinely has to exercise the whole system.