Before you fail a dependency in a resilience test, what must you write down first, and why?

## A fault is an action; it is not a verdict Every test has three parts: a setup, an action, and an **oracle** — the thing that decides whether what happened was right. Resilience testing keeps the first two and routinely loses the third. The action is a deliberate failure of a dependency, a node or a link, and failures are loud: logs fill with stack traces, retries stack up, dashboards turn red. In that noise it is easy to accept "the service stayed up" as the verdict. "The service stayed up" is not a specification of anything. It is satisfied equally by a system that refused the write cleanly and by one that silently did the wrong thing. The fix is cheap and almost entirely non-technical: **before the fault is injected, write down what the system is supposed to do while the dependency is unavailable.** That written statement is the oracle. Everything else — the injection mechanism, the duration, the automation — is plumbing around it. ## What the statement has to contain Write it per *feature*, not per system. "The service degrades gracefully" is unfalsifiable; a system has as many degraded behaviours as it has features touching the broken dependency. For each dependency-and-feature pair, state: - **The response the caller gets** — a refusal with a specific code, or a success built from a reduced data set. - **What the interface shows** — an empty state, a value explicitly labelled as stale, a control that is disabled rather than one that fails on click. - **Which side effects remain permitted** — may it still write? still charge? still emit an event downstream? - **What must never happen** — this is the clause people skip, and it is the one that catches the dangerous defects: no partial charge, no widened rights, no silent success reported to the caller. - **What operators are told** — the specific signal that fires, so that "degraded" is observable from outside. - **How it comes back** — does full behaviour return by itself when the dependency recovers, within what window, and does anything queued during the outage need draining or backfilling? ## Why the order matters Three reasons, and all three are about the order rather than the content. First, it is an oracle problem. Written after the run, the statement is a *description of what you observed*, dressed up as a requirement. Human judgement is extremely good at rationalising an outcome once it has been seen. Second, it forces a decision to be made by someone entitled to make it. Whether a feature fails closed (refuse and say so) or serves something reduced is a product and risk decision. Written first, someone answers it. Left unwritten, it is answered by accident — by whatever a client library happens to do on timeout, or by whichever branch a developer wrote at three in the afternoon. Third, a written expectation converts mechanically into assertions, which means the case is repeatable and can sit in the regression pack instead of being re-argued at each drill. ## A worked example A music-streaming service supports collaborative playlists. Edit rights come from a separate entitlement service. The team's 340-case regression pack contained 11 cases for "entitlement service unavailable", and every one of them asserted the same thing: the playlist page still renders and the service returns something that is not a server error. In production the entitlement service timed out for 26 minutes. The playlist service's client fell back to a permissive default, and any signed-in collaborator was treated as the playlist owner. Followers with view-only rights renamed and deleted other people's playlists — a permission escalation that all 11 resilience cases passed with flying colours, because nothing crashed and the page rendered. Had the expectation been written first, that row would have read: *reads succeed from cached membership; every write returns a refusal with code `ENTITLEMENT_UNAVAILABLE`; no rights are ever inferred when membership is unknown; the refusal is audited; the degraded signal fires.* That is four assertions and a signal check, and the fail-open default would have failed on its first run rather than in production. ## Turning it into assertions Build a small table: rows are dependency × feature, columns are the six items above. Each row becomes one test case that asserts, in order, the response, the persisted state (including **the absence** of the forbidden side effect — the record was not written, the event was not emitted, the audit line exists), the operator signal, and the recovery. Asserting absence is the habit that separates a resilience test from a smoke check, because escalation, double-charge and double-send bugs are all bugs of *extra* behaviour, and no status-code assertion will ever see them. ## Common ways it goes wrong - Treating "degraded" as one system-wide mode instead of a per-feature contract. - Stating the expectation in terms of an internal mechanism — a breaker opening, a cache being consulted — rather than observable behaviour. A test coupled to the mechanism passes whenever the mechanism runs, including when the mechanism is doing the wrong thing. - Asserting only the outage and never the return, so recovery is exercised for the first time during a real incident. - Letting whoever runs the fault also decide, on the day, what a pass looks like.