Circuit Breaker

## What the breaker is, and what CLOSED does A circuit breaker is a **stateful proxy** sitting between a caller and a remote dependency — an HTTP call to another microservice, a database query, a third-party API. It has three states: `CLOSED`, `OPEN`, and `HALF_OPEN`. In `CLOSED`, every call executes for real against the dependency, and each outcome (success, exception, timeout) is recorded into a rolling record of recent calls. The breaker continuously evaluates that record against configured thresholds — most commonly a **failure-rate percentage** — and the moment the threshold is breached it transitions to `OPEN`. ## Why the pattern exists The reason this pattern exists is to stop **cascading failure**. When a downstream dependency starts erroring or hanging, callers that keep hitting it as if nothing is wrong pile up blocked threads, exhausted connection pools, and growing latency, and that pressure propagates upstream: - a slow database can take down the API layer, - which takes down the UI, - which takes down everything depending on the UI. It also hurts the struggling dependency itself: a flood of retrying callers is exactly the load it can least handle while trying to recover. The circuit breaker is named after its electrical namesake: it trips to interrupt the circuit before the damage spreads, rather than letting current (traffic) keep flowing into a fault. ## What happens once it trips Once `OPEN`, calls are short-circuited immediately: no network call is attempted at all, and the caller gets an instant rejection (an exception, or ideally a fallback response) instead of waiting out a slow timeout. The breaker starts a **wait-duration timer** at this point. When that timer elapses, the breaker moves to `HALF_OPEN`, a probationary state where it permits a small, configured number of real calls through as probes while everything else is still rejected. 1. If enough of those probes succeed, the breaker closes and resets its counters to a clean slate. 2. If they fail, it reopens and restarts the wait timer, often with some backoff so it doesn't hammer a still-broken dependency every few seconds. ## The core trade-off The core trade-off is **availability versus protection**. Failing fast during `OPEN` reduces load and latency pressure on both the caller and the dependency, but it also rejects calls that might have succeeded if the dependency had already quietly recovered — you're trading some false rejections for a guarantee against pile-up. Tuning the wait duration is itself a balancing act, and threshold tuning has the same shape: | Knob | Set too tight | Set too loose | |---|---|---| | Wait duration | too short and the breaker flaps between open and half-open under sustained partial failure, hammering a dependency that hasn't fully recovered | too long and you needlessly extend an outage after the dependency is healthy again | | Trip threshold | too sensitive and transient blips (a couple of timeouts during a GC pause) trip the breaker unnecessarily | too lax and a real outage takes too long to trigger protection | ## Failure modes in production In production: - A common failure mode is **only counting hard exceptions** toward the threshold and ignoring calls that succeed but are pathologically slow — a hung downstream that eventually returns 200 after 30 seconds looks 'healthy' to a naive breaker while it quietly exhausts every caller thread. This is why real implementations pair a failure-rate threshold with a separate **slow-call-rate threshold**. - Another common issue is **evaluating the threshold against too small a sample** — three calls at startup, two of which fail, trips a breaker that hasn't seen enough traffic to justify the decision, which is why implementations enforce a **minimum-number-of-calls floor** before they'll even evaluate. ## A concrete example A concrete example: an e-commerce checkout service wraps its call to a payment gateway in a circuit breaker. Under normal operation the breaker stays closed and every checkout hits the gateway directly. If the gateway starts timing out during an incident, the breaker trips open after a handful of failures, and every subsequent checkout attempt immediately gets a 'payments are temporarily unavailable, please retry' message instead of hanging for 30 seconds waiting on a doomed call — protecting the checkout service's own thread pool from being drained by calls to a dependency that isn't going to answer.