A stress test fails roughly once every few thousand runs. How do you turn that into a reproducible, minimal failing schedule you can actually debug?

## Why minimization is the whole game A rare failure carries almost no information in raw form: a stack from thread 7 out of 32, after 41,993 iterations, with several hundred operations of history. You cannot tell which operations mattered. A minimized case — two threads, three operations, one preemption point — usually explains itself on sight. The engineering effort belongs in getting from the first to the second, not in staring at the first. ## Phase 1: make it repeat **Seed everything.** Every random choice in the harness — operation selection, key selection, delay lengths, thread counts — must derive from one per-iteration seed. On failure, log the seed plus the whole configuration (thread count, core count, build flags, instrumented or not). Then provide a way to run exactly that iteration. This single discipline converts many "unreproducible" failures into on-demand ones, because they were never about timing at all — they were about which operations the harness happened to pick. **Log the interleaving.** Have each thread append `(thread, operation, arguments, sequence)` to a preallocated per-thread buffer, merged on failure. This is cheap enough not to distort timing much, and it gives you the actual order of the operations that mattered. Even without exact reproduction, this trace often identifies the pattern. **Escalate to controlled scheduling.** If the failure genuinely depends on timing, move the test under a scheduler you control, where the seed fixes every switch decision. Replay becomes exact, and you can then also *search* schedules deliberately instead of waiting for one. **Raise the rate before you debug.** Shrink the test body, widen the dangerous window with deliberate delays at the suspected point, raise the thread count, run on more cores. Going from 1-in-3000 to 1-in-20 makes every subsequent step cheap. Note that the delays are a diagnostic instrument here, not a fix — they come back out. ## Phase 2: minimize by delta debugging Delta debugging is a general algorithm: given a failing input and a test that says fail-or-not, find a minimal failing input. Model your execution as a list of elements and apply it. What the elements are, in order of payoff: 1. **Operations.** Drop each operation from the recorded sequence; keep the drop if the failure survives. 2. **Threads.** Try removing whole threads. Most bugs are pairwise; going from 16 threads to 2 is often the biggest single reduction. 3. **Delay/preemption points.** Under a controlled scheduler, remove injected preemptions one at a time; what remains is the *preemption depth*, and it is usually 1 or 2. Knowing exactly which two points must be preempted is often the entire root cause. 4. **Data.** Collapse distinct keys and values to the smallest set that still fails — frequently one key, revealing the contention point. 5. **Configuration.** Reduce buffer sizes, capacities and timeouts to their smallest failing values. Run the reduction loop automatically. Each candidate must be evaluated by re-running; if reproduction is probabilistic rather than deterministic, run each candidate `k` times and treat "failed at least once" as failing — with the caveat that a candidate can be wrongly discarded, so use a generous `k` and accept an approximately-minimal result. Stop at **1-minimality**: removing any single remaining element makes the failure vanish. ## Interference to expect - **Heisenbug behaviour**: logging and instrumentation change timing and can hide the failure. Prefer preallocated lock-free buffers and dump only on failure; if the bug still evaporates, that itself is evidence about how tight the window is, and is a reason to move to a controlled scheduler where timing is not the mechanism. - **Non-monotonic reduction**: removing an operation can occasionally *change* the failure into a different one. Verify the minimized case fails the same way — same invariant, same location — not merely that it fails. - **Environmental dependence**: if it only reproduces on one machine shape, record that as part of the case rather than chasing a universal reproduction. ## What to do with the result Commit the minimized case as a deterministic regression test, seed and all, next to the code it exercises. Write down the schedule in a comment — "fails when thread B is preempted between the read and the compare" — because that sentence is the actual knowledge produced. Then fix the code, and confirm the minimized case now passes while the full stress harness continues to run with fresh seeds; a fix that only satisfies the minimized case has not been checked against the general problem.