When designing a system, how do you decide between fail-fast and fail-safe (graceful degradation) behavior, and what are the trade-offs?

## Two strategies, two purposes - **Fail-fast**: on a problem, **stop immediately** and surface it (throw/abort). Best when continuing would corrupt data or hide a bug. Optimizes for **correctness and fast debugging**. - **Fail-safe** (a.k.a. fail-soft / graceful degradation): on a problem, **keep operating in a reduced or fallback mode** instead of crashing. Optimizes for **availability and resilience**. (Note: 'fail-safe' is overloaded. In Java collections it means a non-throwing iterator. In systems design it means graceful degradation. This question is the systems-design sense.) ## The decision rule: classify the failure The core skill is **classifying each failure mode**: 1. **Programmer error / broken invariant** (a `null` that should never be null, an impossible enum value, a corrupted internal data structure). → **Fail-fast.** This is a bug; recovering silently just spreads corruption and ships a defect. Throw, alert, and fix. 2. **Expected, external, transient condition** (downstream timeout, rate limit, network blip, optional feature down, cache miss). → **Fail-safe.** These *will* happen in production no matter how correct your code is; the system should absorb them and stay useful. A useful heuristic: *if the failure means my code is wrong, fail fast; if it means the world is temporarily uncooperative, fail safe.* ## Fail-safe mechanisms (resilience patterns) - **Retries with backoff** — for transient faults; cap attempts, add jitter. - **Circuit breaker** — after repeated failures, stop calling a sick dependency for a cooldown, returning a fast fallback; prevents cascading failure and resource exhaustion. - **Fallback / default values** — serve cached/stale data or a sane default when the live source is unavailable. - **Bulkheads** — isolate resources (thread pools, connection pools) per dependency so one failing dependency can't sink the whole service. - **Timeouts** — never wait forever; a missing timeout turns a slow dependency into a full outage. - **Graceful degradation** — turn off non-essential features (recommendations, analytics) while keeping the core path alive. ## The non-negotiable: fail loudly to observability The biggest trap is conflating **fail-safe** with **fail-silent**. Degrading for the *user* is fine; hiding the failure from *operators* is not. Every degraded path must emit logs, increment metrics, and (for important ones) trigger alerts. Otherwise you accumulate silent partial failures and discover them only via customer complaints. Fail-safe = *visible* recovery, never invisible. ## Trade-offs to articulate | Dimension | Fail-fast | Fail-safe | |---|---|---| | Optimizes for | correctness, fast debugging | availability, resilience | | Risk if misapplied | crashing on recoverable issues; poor UX | masking real bugs; serving wrong data | | Right for | internal invariants, dev/test, data integrity | external deps, transient faults, production UX | | Cost | downtime on edge cases | complexity, possible stale/partial results | ## Where they combine A mature service does **both at once**: fail-fast internally (strict validation, no swallowing of internal exceptions) and fail-safe at the integration edges (circuit breakers, fallbacks). Example: an e-commerce checkout fails fast if the *order total is internally inconsistent* (a bug), but fails safe if the *recommendations service* is down (just hide recommendations). And money/safety-critical paths often deliberately fail-fast even in production — better to reject a transaction than to process a possibly-corrupt one. ## Money/safety-critical nuance For financial, medical, or safety-critical operations, correctness usually outranks availability: prefer **fail-fast / fail-stop** (refuse rather than guess). 'Fail-safe' there can even mean failing *into a safe state* (e.g. brakes engage on power loss) — a third meaning worth naming if relevant.