Retry and Backoff

## Why a tight retry loop makes the problem worse When a call to a remote dependency fails, the naive response is to retry right away, and if that fails, retry right away again. This **busy retry** pattern is a mechanism problem before it's anything else: each failed attempt still consumes - a TCP connection, - a thread or async task slot, - TLS handshake CPU, - and a slot in the target's own connection-accept queue or thread pool. If the reason for the failure is that the downstream service is overloaded or degraded — the single most common cause of transient errors in production — hammering it with immediate retries adds load at exactly the moment it has the least spare capacity to absorb it. Multiply this by the fact that a single client-side failure (say, a database blip) is usually seen by many callers at once — hundreds of service instances, or thousands of end-user clients — and a tight retry loop turns one brief hiccup into a synchronized wave of demand that can push a recovering service back into failure, a feedback loop sometimes called a **retry storm**. ## What exponential backoff changes Exponential backoff addresses this by making each successive retry wait longer than the last, with the delay growing geometrically rather than staying constant (fixed-interval retry) or growing linearly. The canonical formula is `delay = base * factor^attempt`, e.g. `base=1s`, `factor=2` gives 1s, 2s, 4s, 8s, 16s... The delay is almost always - **capped at a maximum** (say 30s or 60s) so it doesn't grow unboundedly, - and paired with a **maximum attempt count** or an overall **deadline** so a caller doesn't retry forever and hang the calling code path. ## The core trade-off: latency versus load The core trade-off is latency versus load: a caller that backs off aggressively takes longer to eventually succeed (worse for the end user if the dependency does recover quickly) but is much gentler on a struggling dependency and far less likely to contribute to an outage. A fixed short interval (e.g., always retry after `200ms`) gives faster recovery if the failure was a one-off blip, but if the failure is systemic — the dependency is actually overloaded — a fixed-interval retry keeps re-adding load at a constant, non-decreasing rate and never gives the system room to drain its backlog and recover, so error rates can plateau instead of resolving. ## Failure modes in production Failure modes in production typically show up as one of two shapes. 1. **The first is naive retry without any backoff:** dashboards show request volume to a dependency several multiples higher than the legitimate call rate, latency percentiles balloon, and the dependency never gets a chance to recover because retries keep re-filling the queue it's trying to drain — a classic **retry amplification** incident. 2. **The second is backoff without a cap:** the delay grows so large after a few attempts that the caller effectively gives up in practice (a 64s or 128s wait is often longer than the caller's own timeout budget), so an overly aggressive exponent can silently turn retries into de-facto single attempts. A related, subtler failure is applying exponential backoff on the client side of a multi-hop call chain without matching the backoff policy against the overall request deadline: if a user-facing request has a 3-second SLA, and a downstream client independently retries with 1s, 2s, 4s backoff, the third retry can't complete before the caller has already timed out and given up — meaning the backoff schedule needs to be designed against the actual deadline budget, not chosen in isolation. ## Where the same shape shows up A concrete real-world instance of this is how AWS SDKs describe client retry behavior for services like DynamoDB and S3: the default retry policy is exponential backoff with a capped maximum delay, specifically because AWS's own operational experience showed that naive or fixed-interval retries from thousands of independent SDK clients were a meaningful contributor to prolonging service-side incidents rather than helping clients route around brief ones. The general principle generalizes to any client library talking to a shared, rate-limited, or capacity-constrained backend: - HTTP clients retrying 5xx responses, - message-queue consumers reprocessing failed messages, - or gRPC clients on transient `UNAVAILABLE` status codes. All apply the same `base * factor^attempt` shape, and the parameters (base delay, factor, cap, max attempts) are a genuine design decision, not a boilerplate default to leave untouched.