Resilience & Observability

## What the breaker wraps and what it counts A **circuit breaker** is a stateful wrapper placed around every outbound call a service makes to a particular dependency (a database, another microservice's HTTP endpoint, a third-party API). Concretely, it holds a **rolling window** of recent call outcomes - counted either by a fixed number of calls (e.g., the last 20) or by a time window (e.g., the last 10 seconds) - and classifies each outcome as: - **success**; - **failure** - an exception, connection error, or configured HTTP status like 5xx; - or, in more advanced implementations, **"slow call"** - a call that succeeded but exceeded a latency threshold. ## The three states The breaker itself is a three-state machine. 1. In the `CLOSED` state, everything behaves normally: calls pass straight through to the dependency and their outcomes feed the rolling window. Once the failure rate (or slow-call rate) in that window crosses a configured threshold - and only after a minimum number of calls have been observed, so a handful of early failures on a cold window doesn't trip it prematurely - the breaker transitions to `OPEN`. 2. In `OPEN` state, calls are rejected immediately, in-process, without ever touching the network: the caller gets an exception or a fallback value in microseconds instead of waiting out a timeout. 3. After a configured wait duration, the breaker moves to `HALF_OPEN`, where it allows a small, limited number of "trial" calls through to test whether the dependency has recovered; if enough of those succeed it transitions back to `CLOSED`, and if they fail it reopens for another wait period. ## Why the pattern exists The reason this pattern exists is to stop a single struggling dependency from taking down the caller, and by extension everything upstream of the caller. In a synchronous microservices architecture, a slow or hanging downstream call ties up the caller's own resources - threads, connections, memory buffering the in-flight request - for as long as the caller is willing to wait. If that dependency is degraded rather than fully down, every retry and every default socket timeout compounds the problem, because the caller keeps issuing new calls that also hang, exhausting its own thread pool. Once a service's thread pool is exhausted handling calls to one bad dependency, it can no longer serve requests that don't even touch that dependency, and its own callers start timing out waiting on it - and the failure cascades up the call graph. The circuit breaker breaks this chain by making the caller **fail fast** once it has enough evidence the dependency is unhealthy, freeing its own resources immediately instead of spending them waiting. ## The trade-off The trade-off is real: - An **aggressive threshold** protects the caller quickly but risks tripping on transient blips - a single slow GC pause or a momentary network hiccup - and then serving degraded behavior to users who would have been fine with a real call. - A **lax threshold** avoids false positives but lets pressure build for longer before the breaker actually helps, during which the caller's own resource exhaustion may already be underway. Threshold tuning (failure-rate cutoff, window size, minimum call count, wait duration) has to be calibrated per dependency based on its normal error rate and latency profile - there's no universal setting. A second, often-missed trade-off is that a circuit breaker requires you to decide what happens when it's open: some **fallback behavior** (cached/stale data, a degraded feature, a queued-for-later write, or simply a fast, clear error) has to exist, or the breaker just converts a slow failure into a fast one without actually improving the user's experience. ## Production failure modes Production failure modes cluster around a few patterns. 1. **First, misclassifying faults**: if the breaker's classifier counts expected client-side responses (like a 404 for a genuinely missing resource) as failures, it can trip for reasons that have nothing to do with the dependency's actual health - typically only 5xx responses, timeouts, and connection errors should count. 2. **Second, "flapping,"** where the dependency is right on the threshold boundary and the breaker cycles between `OPEN` and `HALF_OPEN` repeatedly, adding instability and confusing alerts. 3. **Third, granularity mismatches**: a breaker configured per-service when the service exposes many endpoints of very different reliability can trip broadly for a problem confined to one endpoint. 4. **Fourth**, at fleet scale, if every instance's breaker opens around the same time and their half-open cooldowns are synchronized, all instances can send trial calls simultaneously when the wait expires, re-tripping the dependency right as it was starting to recover - a thundering-herd problem usually fixed with jitter on the cooldown timer. ## Where you meet it Netflix popularized this pattern at scale with Hystrix; the more current standard in the JVM ecosystem is `resilience4j` (commonly used via Spring Boot's resilience4j integration), and infrastructure-level equivalents exist too - Envoy/Istio's "outlier detection" implements similar logic at the service-mesh layer, ejecting unhealthy endpoints from the load-balancing pool without any application code.