A Cucumber-JVM pack outgrew the pipeline budget — how do you choose between tag tiering, parallelism and sharding?

## Decide what you are protecting before you pick a lever A pack of 214 scenarios that takes 31 minutes against a 12-minute pull-request budget is not one problem. It is three questions that people collapse into one: 1. what must be true **before a change merges**; 2. what may be discovered **an hour later**, on the main branch; 3. what may wait **until tomorrow**, in a nightly run. Every lever below is really an answer to that question, and choosing a lever without answering it first is how teams end up with a fast pipeline that catches nothing. So begin with measurement. Per-scenario duration, not total wall clock — the total tells you the pack is slow, the distribution tells you whether eleven scenarios own half the time or whether all 214 are uniformly mediocre. Those two shapes have different answers, and the second one is much worse news. ## The four levers and what each one costs | Lever | Buys | Costs | Wrong when | |---|---|---|---| | Tag tiering | immediate feedback on a subset | coverage between stages | the untagged remainder is where regressions actually live | | In-JVM parallelism | the same coverage in less time | parallel-safe glue, load on the system under test | the suite has shared state you have not fixed | | Sharding across agents | scale beyond one machine | partitioning, aggregation, infrastructure spend | the pack is slow because of a few scenarios, not its size | | Moving assertions down | a permanently smaller pack | time, and a team willing to delete scenarios | the behaviour genuinely only exists end-to-end | **Tag tiering** is where most teams start, because it is a text edit. A 7-minute tagged tier on every push and the full pack on merge is a defensible split — as long as you say out loud what you have accepted: a regression outside the tag reaches the main branch and is found by a later stage. That is a risk decision, not a technical one, and it belongs to whoever owns the release. The failure mode is tag rot: nobody adds new scenarios to the fast tier, it thins out over two quarters, and the gate becomes ceremonial. **Parallelism** keeps all the coverage, which is why it is the most attractive lever, and it demands the most from the glue: no shared static state, no scenario depending on another's leftovers, per-scenario test data. It also relocates the bottleneck. Six threads against one test database or one browser grid slot produces timeouts that read like flakiness, and the temptation is then to add retries, which buys back the runtime you just saved. **Sharding** is the lever that scales furthest and the one people underestimate. Splitting a pack across six agents means owning a partition rule — by feature file, by tag, or, best, by recorded duration so the shards finish together — plus an aggregation step that turns six partial results into one verdict and one report. It also changes triage: a failure now lives in one shard's log, and reproducing it locally means knowing which shard it ran in. **Moving assertions down the pyramid** is the only lever that makes the number smaller instead of hiding it. A scenario that asserts a pricing rule the domain layer can assert directly does not belong in an end-to-end pack. This is slow, unglamorous work, and it is the difference between a suite that stays viable for three years and one that gets rewritten. ## A worked decision On the vinyl-record marketplace pack: measurement shows 214 scenarios, 31 minutes, and a long tail — 38 browser-driven scenarios account for 19 of those minutes. A browser upgrade lands mid-sprint and eight of those 38 turn intermittent for a week. That shape argues for a specific combination. Keep the gating tier off the browser entirely: a 7-minute tagged tier of service-level scenarios that a browser upgrade cannot destabilise, so the gate stays trustworthy exactly when the environment is moving. Run the full pack on merge with modest fixed parallelism, sized against the test database rather than the agent. Shard only if merge-stage time still misses its budget after the tail is addressed — because sharding a pack whose slowness is concentrated in 38 scenarios buys much less than fixing those 38. ## How you know it worked Speed is the easy metric and the misleading one. Track alongside it: - **escaped regressions per stage** — how often something the fast tier could have caught was caught later instead; - **retry and flake rate**, because a suite that halved its runtime and doubled its retries is slower in practice; - **shard skew**, the gap between the fastest and slowest shard, which is the efficiency of your partition rule; - **agent minutes per merge**, the bill the sharding decision creates; - **the fast tier's share of the pack**, watched over quarters, so tag rot is visible before it is fatal. And make it a standing rule rather than a project: every new scenario declares which tier it belongs to at review time, and the reviewer is entitled to ask whether it needs to be a scenario at all. That convention is what keeps the budget from being renegotiated every six months.