Your recursive log parser passed every test, then died at 3 a.m. on a million-record chain — why was that inevitable?

## Why "we got unlucky" is the wrong diagnosis The skeptic's version of this incident is that a rare, oversized input showed up and tipped the parser over. The accurate version is that the parser's stack consumption was a **deterministic function of input length** — one frame per record — and production simply supplied an input on the other side of a fixed threshold. Nothing was random. Depth = number of records; capacity ≈ stack region bytes / frame bytes. Once those two numbers were set, the crashing input size was determined, and everyone was merely waiting for traffic to reach it. That reframing matters, because "unlucky" invites a retry and a bigger region, while "deterministic" invites the question that actually resolves the incident: *what is the largest chain we can accept, and who guarantees it?* ## Why the test suite could not have caught it Three separate reasons compound: 1. **Fixtures were three orders of magnitude too small.** Thousand-record fixtures exercised roughly a thousand frames. Production needed a million. No amount of *repeating* those tests explores the axis that mattered — this is a scale property, and repetition is not scale. 2. **The failure is discontinuous.** Allocation-heavy problems usually announce themselves: memory use climbs, collection work rises, latency drifts up. Stack consumption produces none of that. Pushing the ten-thousandth frame costs the same as pushing the tenth. The system is perfectly healthy right up to the frame that does not fit, and then the execution context is gone. There is no curve to extrapolate from a small run. 3. **Everything the tests measured looked fine.** Correctness was fine — the algorithm is right. Throughput was fine — the work is linear. Allocated memory was fine — the routine allocates nothing. The one resource being consumed proportionally to input was the one nobody instrumented. The general lesson is worth saying out loud in an interview: **a test suite only protects the resources it observes at the sizes it exercises.** ## Why 3 a.m. and not during business hours The distribution of chain lengths usually has a long tail driven by batch or backlog behaviour — an overnight consolidation job, a retry storm, a slow consumer that let a chain accumulate. The largest inputs a system ever sees are typically produced by machines during quiet hours, not by users during peak. So the first crossing of a size threshold characteristically happens when nobody is watching, and that is a reason to bound the input rather than to hope. ## What to change, in the order a reviewer wants to hear it **1. Name the depth function.** For every recursion over externally supplied data, write down depth as a function of input: constant, logarithmic, or linear. Linear depth over unbounded input is a defect at design time, not a surprise at runtime. This single habit prevents most repeats. **2. Test at the documented bound, not at the convenient size.** If the contract says chains may reach one million, the suite must run one million at least once, even if that test is slow and lives outside the fast suite. If nobody can state the bound, that is the real finding of the postmortem. **3. Instrument peak depth.** Depth is cheap to track — a counter incremented on entry and decremented on return, with a high-water mark reported per run. That converts an invisible cliff into a graph you can alarm on well below the ceiling, which is the difference between a ticket next week and a page at 3 a.m. **4. Fail the input, not the process.** Enforce an explicit depth or size limit and reject the oversized chain with a clear diagnostic. A rejected record with a named reason is vastly better operationally than an aborted execution context that takes unrelated in-flight work down with it, and it also closes an availability risk: if an outsider can influence chain length, an unbounded recursion is a denial-of-service surface, not merely a robustness gap. **5. Remove the coupling if the input is genuinely unbounded.** Bounding depth so it no longer tracks input size is the only change that retires the failure class rather than relocating the threshold. Enlarging the region is a stopgap: a constant factor against an unbounded quantity, acceptable overnight with an alarm attached, embarrassing as a permanent answer. ## The postmortem sentence that lands "Our stack usage was linear in an input we do not control, we never measured it, and the failure mode gives no warning — so the crash date was set the day we shipped it." That sentence gets agreement from the skeptic, because it makes no claim about luck and identifies the missing control precisely.