Once a circuit breaker's wait duration expires and it enters half-open, how does it decide whether to fully close again or trip back to open, and why does it deliberately limit how many calls are allowed through during that phase instead of just resuming full traffic?

## A deliberately constrained state Half-open is a deliberately constrained **probationary state**, and understanding why it's constrained the way it is matters more than memorizing that it exists. When the open-state wait-duration timer expires, the breaker doesn't simply flip back to closed and let all traffic resume — it transitions to half-open and permits only a small, explicitly configured number of calls through (a 'permitted number of calls in half-open state', commonly single digits to low tens) while every other concurrent request during this window continues to be short-circuited exactly as in open state. ## How the trial probes are judged Those permitted calls are **trial probes**: real calls to the real dependency, with real outcomes recorded. The breaker evaluates those outcomes against essentially the same kind of threshold logic used in closed state — a failure-rate and/or slow-call-rate percentage — just computed over this small trial population instead of the full sliding window. - If the trial calls meet the success bar, the breaker transitions to **closed** and resets its statistics to a clean slate, resuming full unrestricted traffic. - If the trial calls fail to meet the bar, the breaker transitions back to **open**, restarting the wait-duration timer (sometimes with backoff, so repeated failed recovery attempts wait progressively longer between tries). ## Why the trial count is capped The reason the number of trial calls is capped rather than just resuming full traffic is **risk containment**. A dependency can be in one of several states when the wait duration expires: 1. fully recovered, 2. partially recovered (able to handle some load but not full production volume yet), 3. or still down. If the breaker dumped its entire backlog of accumulated, pent-up production traffic onto the dependency the instant the timer expired, a partially-recovered dependency could be knocked straight back into failure by the very traffic surge the breaker was protecting it from — the breaker would effectively cause the exact re-trip it exists to avoid, and worse, do so with a burst rather than a controlled trickle. Limiting the trial to a handful of calls means the breaker tests recovery cheaply: if the dependency is still broken, only those few calls pay the cost of finding out, not the entire caller population. ## The trade-off in sizing the probe The trade-off is **speed of detection versus safety of the probe**. - A very small trial count (say, 1-2 calls) is the gentlest possible test but is statistically noisy — a single unlucky timeout on an otherwise-recovered dependency sends the breaker straight back to open, extending the outage from the caller's perspective for another full wait-duration cycle even though the dependency was fine. - A larger trial count gives a more reliable signal about true recovery but exposes more of the (possibly still-fragile) dependency to load during the probe, and takes longer for all trial calls to complete before a decision can be made, especially if the dependency is still slow. Tuning this number is the same kind of statistical-noise-versus-responsiveness trade-off seen in threshold tuning generally. ## The thundering-herd-on-recovery failure mode A production failure mode specific to half-open is what's sometimes informally called the **thundering-herd-on-recovery** problem: if a service is deployed as many replicas, each running its own in-process breaker independently against the same shared downstream dependency, those breakers tend to trip around the same time under a shared incident and their wait-duration timers, started at roughly the same moment, expire at roughly the same moment too. Dozens or hundreds of replicas can then simultaneously enter half-open and simultaneously send their permitted trial calls at the exact same instant, effectively reconstituting the very load spike that caused the outage, just synchronized. Mitigations include: - **jittering the wait duration** per instance so timers don't expire in lockstep, - or **moving circuit breaking to a shared layer** (like a service mesh sidecar) that can coordinate state across replicas rather than each instance probing independently. ## A concrete scenario A concrete scenario: a search service's breaker trips against a recommendations backend that's been overloaded and returning timeouts. After a 30-second wait duration, the breaker allows 5 trial calls through. If all 5 succeed within acceptable latency, the breaker closes and the search service resumes calling recommendations normally for all subsequent requests. If 3 of the 5 still time out, the breaker reopens, waits another 30 seconds (or a backed-off longer interval), and tries again — protecting the recommendations backend from a full traffic surge while it's still digging itself out, and protecting the search service's own threads from being re-exhausted by a premature full resumption.