A full mutation run takes six hours nightly — how do you make it pay for itself?

### Framing the problem A six-hour full-repository mutation run is not, on its own, a failure. It is a signal that the technique has been deployed at the wrong granularity for the feedback loop it is supposed to serve. The question a lead has to answer is not "how do we make the run faster" but "what decision is this run supposed to inform, and what is the cheapest run that informs it?" Everything below follows from that. Mutation testing produces two distinct goods, and they want different cadences: - **Per-change evidence** — "the tests you just wrote actually check the code you just changed." This must land inside a review cycle, so minutes, not hours. - **Portfolio evidence** — "here is where the suite is structurally weak across the codebase." This is a planning input consumed weekly or monthly, and a long batch run serves it fine. Conflating them is the usual mistake: a team points the whole-repository run at the pull-request gate, engineers wait, the gate gets muted, and the technique is abandoned as impractical. ### Cut the work, in order of leverage **Restrict the mutant population to the change.** Generate mutants only on lines added or modified relative to the merge base, and run only the tests that reach them. On a typical change this takes a run from thousands of mutants to tens. This is the single change that makes the technique viable in review, and it aligns with how teams already treat new-code coverage. **Select tests per mutant.** A mutant can only be killed by a test that executes the mutated line, so the runner should use a coverage map to pick that subset instead of running the full suite for every mutant. Most of the naive cost is here. **Cache across runs.** A mutant on unchanged code, killed by an unchanged test, has a known answer. Persisting results between runs and re-evaluating only the affected region turns the nightly job into an incremental one. **Tune the operator set.** The default set is usually a good tradeoff; the extended set multiplies both run time and the equivalent-mutant triage burden. Start narrow. **Sample when you must.** For portfolio evidence a random sample of mutants gives a usable estimate of the score at a fraction of the cost. Sampling is inappropriate for a gate — the developer needs the specific survivor on their line, not an estimate. **Parallelise and scope.** Mutant evaluation is embarrassingly parallel. Splitting by module also lets you run high-risk modules more often than the rest. ### Then decide what it gates My default: mutation testing is **advisory on the pull request and blocking nowhere at first**. The runner posts survivors on the changed lines as review comments; the human reviewer decides. After a few months, when the team has internalised what a survivor means, a narrow gate becomes defensible — for example, no *new* survivors on changed lines in a small set of modules where a missed fault is expensive. A global score threshold is the worst of the options: it is capped by equivalent mutants, it moves when unrelated code moves, and it invites gaming through operator exclusions. The risk-weighting matters more than the threshold. In the grant-application review queue, the reviewer-eligibility guard is worth mutating on every change, because a survivor there is the signature of a permission escalation — a reviewer approving their own application. The report renderer is not; a survivor there costs a cosmetic defect. Concentrating a scarce, expensive signal on the code where an undetected fault is expensive is the actual strategy; run-time tuning is plumbing in service of it. ### Sequencing the adoption 1. **Baseline once.** Run the full analysis on a weekend to learn the true score, the equivalent-mutant share, and which modules are weakest. That is the six-hour run, and you need it exactly once before you decide anything. 2. **Pick two or three high-consequence modules** from the baseline and turn on changed-lines analysis for them in review, advisory only. 3. **Budget the triage.** Survivors need people. If nobody owns the queue the run becomes noise within a month. Fold triage into code review rather than making it a separate ritual. 4. **Keep a periodic full run** — weekly, off-hours, incremental — as the portfolio signal, and watch the trend rather than the absolute number. 5. **Expand by evidence.** Extend to more modules when the advisory signal has demonstrably caught things review missed; retreat where it produces mostly equivalents. ### What to say about cost honestly Mutation testing is the most expensive routine quality signal a team can adopt, in machine time and in human triage. It is worth it where a missed fault is expensive and cheap coverage numbers are already high enough to be uninformative. It is not worth it as a blanket mandate, and a lead who proposes it as one should expect to be asked what the team stops doing to pay for it.