You are scoping the first production chaos experiment for a payment service. Along which dimensions do you minimize the blast radius, and how do you decide when to widen it?

## Why the blast radius is the whole safety argument An experiment is only worth running if it might falsify your hypothesis — which means it might hurt. The blast radius is what converts "might hurt" into "hurt an amount we chose in advance". Every other safeguard is secondary: abort conditions limit how long the damage lasts, but the blast radius limits how much damage per unit time is even possible. ## The four dials **Population.** The strongest dial, and the first to reach for. Options, roughly in order of increasing exposure: an internal or synthetic client only; employees or a beta cohort; one tenant on a multi-tenant platform; one shard or partition; one instance out of N; a percentage of live traffic. If the service runs with N+1 or N+2 redundancy, removing one instance is by definition something the design already claims to tolerate — which is precisely the claim worth testing first. **Duration.** A ten-minute run with a hard, automatically enforced cap. Duration is often the cheapest dial to widen later, and the one people most often forget to bound: an experiment left running because everyone got distracted is a self-inflicted incident with no owner. **Severity.** The same fault type has a dial inside it. 100 ms of added latency is a different experiment from 2 seconds; degrading a dependency is different from blackholing it. Start at the mildest setting that could still move the steady-state metric. **Reversibility.** A dial with essentially one acceptable setting. The fault must be removable by a single action, and that removal must be tested before injection. Faults that destroy data, move money, or send real notifications to real customers are not chaos experiments; they are damage. Time of day is not a fifth dial so much as a precondition: business hours, owning team present, not during peak season, not mid-launch, not during a change freeze, and not while an unrelated incident is in progress. ## The floor: small enough to be safe, large enough to be readable Minimising blast radius has a lower bound that people miss. If you expose 0.05% of traffic for two minutes and report "no impact detected", the honest reading is usually *this experiment could not have detected the impact*, not *there is no impact*. The signal has to clear the noise in your steady-state metric. So the rule is the smallest scope that could **still falsify the hypothesis**. When the population you can safely expose is too small to produce a readable signal, the usual moves are to extend duration rather than population, to narrow the steady-state metric to just the exposed cohort so its denominator is small, or to run against a concurrent control group so the comparison is like-for-like instead of against a noisy global baseline. ## Widening: one dial per run After a pass, expand deliberately: 1. Population, in steps — 1% → 5% → 25% → one full zone. 2. Then duration. 3. Then severity. One dial at a time, because failures are frequently **nonlinear**. Killing one instance out of six proves nothing about killing three, because the remaining capacity crosses a threshold: connection pools saturate, autoscaling reacts too slowly, a shared downstream that absorbed the redirected load at 1% collapses at 25%, caches that stayed warm go cold. If you widen two dials at once and the run fails, you cannot attribute the failure. Before each widening, apply the honest test: *if this run falsifies the hypothesis at this new scope, am I willing to have caused that impact?* If the answer is no, you are not ready to widen — you are gambling. ## The programme-level ceiling Beyond the single run, decide in advance what fraction of the service's error budget the chaos programme may consume in a period, and treat it as a hard ceiling rather than a guideline. That converts an unbounded appetite for confidence into a number the product owner can agree to, and it gives you an unambiguous rule for when to stop: if the budget is already spent on real incidents, the experiments pause. It also makes the tradeoff visible in the right direction — an experiment that finds a genuine weakness usually saves far more budget than it spends, but only if the spending is deliberate. ## For the payment service specifically Payment is the case where reversibility dominates everything. Do not inject into the money-movement path itself on the first run. Scope to a read-side or an ancillary dependency — the fraud-scoring call, the receipt service, the rate lookup — assert a hypothesis about checkout success rate, and pick a fault the payment provider will never see. Then widen only when you have evidence, from the earlier runs, that the failure handling around the critical path behaves as documented.