Fault Injection Techniques

## The ordering principle The question behind "what would you inject first" is not "what is most dramatic" but **where is the largest untested assumption, and what does it cost to be wrong there**. For a payment API, being wrong costs money and trust, so both parts of that sentence matter: you go after big unverified claims, and you scope each experiment so a discovered defect is a small event rather than a large one. Rank candidate faults by two axes: how likely the mechanism is to be exercised in a real incident, and how confident you are today that it works. Anything the platform already exercises daily scores near zero on the second axis, no matter how alarming it sounds. ## First: dependency faults, one downstream at a time Twelve downstreams is the story here. Almost every team can name which of them are "critical" and which are "nice to have" — and almost no team has verified the second list. Blackhole one dependency at a time (drop its traffic silently rather than returning an error, because a hang is the more punishing and more realistic fault) and assert the hypothesis explicitly: *a payment still completes successfully, within the normal latency budget, with the non-critical dependency unreachable*. The classic findings are worth naming in an interview because they are what actually happens: - An "optional" fraud-signal, analytics, or personalization call turns out to be synchronous on the request path with no timeout, so an optional dependency can take payments to zero. - The fallback exists but is never reached, because the exception type thrown on a socket hang is not the one the catch block handles. - The call is properly optional but its failure is counted as a payment failure, so an outage in a cosmetic service burns the payment SLO's error budget. Run this for all twelve, not just the interesting ones. The value is in the ones you were confident about. ## Second: latency on the critical few Having established what is truly required, inject delay — not errors — into the required dependencies. Errors return fast and stay contained in one request; latency holds a thread and a connection, and by `concurrency = throughput x latency` a 20x slowdown demands 20x the concurrent slots. That is how a single slow dependency saturates a shared pool and stalls endpoints that never call it. Inject at a value just above the configured timeout so the timeout path, the retry policy, and the fallback all actually fire, and inject on a fraction of calls as well as on all of them, since a single degraded replica behind a load balancer is more common than a clean outage. ## Third: lose an availability zone This is the highest-value structural experiment and the one instance kill can never approximate, because the fault is **correlated**. Three zones, one gone, means the survivors go from carrying roughly 33% of traffic each to roughly 50% each — a 1.5x step change applied instantly. If each zone was running at 70% of capacity, the survivors need 105%, and you have converted a zone failure into a total outage. Sizing for single-zone loss across three zones means keeping steady-state utilization under about two-thirds; that is what N+1 redundancy means in concrete numbers here. Zone loss also probes things nothing else does: whether a stateful dependency's replicas were actually spread across zones rather than co-located by an unnoticed scheduling constraint, whether cross-zone calls exist that you believed were zone-local, whether autoscaling can acquire capacity in the surviving zones quickly enough, and whether anything holds a hardcoded zone-specific endpoint. Inject it by cutting network reachability to the zone rather than by deleting resources — it is more faithful to a real zone impairment and it is reversible in one step. ## Fourth: stateful failover Fail over the database primary, or force a leader re-election in whatever holds consensus. This is the fault with the longest tail of surprises: connection pools that cache a dead endpoint until the process restarts, a failover that takes 40 seconds while callers time out at 2, writes accepted during the window and lost, and application code that reconnects but never re-issues the transaction it was in the middle of. For a payment system this is the one where correctness, not just availability, is at stake. ## Last: random pod termination On Kubernetes, pods are already terminated constantly — every rolling deploy replaces every pod, node autoscaling drains hosts, and preemption reclaims capacity. You pass this experiment daily whether you run it or not, so a deliberate pod kill mostly confirms something already continuously verified. It is not worthless, but it is a low-yield place to spend your first quarter of a chaos program, and choosing it first is the tell that a candidate has read about chaos engineering rather than run it. ## Doing this on a payment path without breaking payments Two practical commitments belong in the answer. Every injection must be **revocable in one action and self-expiring** — a rule that outlives the agent that applied it is an incident, so give it a TTL. And you scope the first run of each experiment narrowly: a subset of traffic or a single zone's replicas before all of them, with the owning team watching live. That is not timidity; on a money path it is what makes running in production defensible at all.