How do you compose timeouts, retries, and a circuit breaker on a WebClient call correctly, and what ordering and idempotency concerns govern the design?

**Goal:** make one remote call robust without amplifying failures. Each mechanism guards a different failure mode; combined incorrectly they fight each other. **The recommended layering (inside → out):** 1. **Per-attempt timeout.** Set transport `responseTimeout` on the Reactor Netty `HttpClient` and/or a Reactor `.timeout(D)` *below* `retryWhen` so each individual attempt is bounded. This ensures a single slow attempt can't consume the whole budget and lets a retry actually happen. 2. **Retry (bounded, filtered, backed off).** `retryWhen(Retry.backoff(maxAttempts, minBackoff).maxBackoff(...).jitter(0.5).filter(transientOnly).onRetryExhaustedThrow((s,sig)->sig.failure()))`. Retry only transient, idempotent failures. 3. **Circuit breaker.** `transformDeferred(CircuitBreakerOperator.of(cb))` placed *outside* the retry so the breaker records the *final* outcome after retries. If the upstream is hard-down, the breaker trips OPEN and short-circuits — which also stops the retry machinery from piling on. 4. **Bulkhead / concurrency limit** (optional but valuable): `BulkheadOperator` caps simultaneous in-flight calls so one slow dependency can't exhaust the connection pool / event loop. 5. **Fallback (outermost):** `onErrorResume(...)` returns a cached value, default, empty, or fast domain error — so callers get a graceful degrade rather than a raw exception. **Ordering rationale:** - **Timeout inside retry** → bounds each attempt. Timeout *outside* retry bounds the whole sequence — use that only as an overall deadline cap, and be aware it can cut off legitimate retries. - **Breaker outside retry** → the breaker must *see* failures. If retries sat outside the breaker, they'd absorb the failures and the breaker would never trip. Conversely with the breaker outside, once OPEN it prevents retries entirely. - Resilience4j's documented composition order for its operators is Bulkhead → TimeLimiter → RateLimiter → CircuitBreaker → Retry (Retry outermost in *their* decorator model), but note their 'outermost' decorator is the *first to be entered*; the practical invariant is unchanged: the breaker's recorded outcome should reflect the retried result. Validate the exact semantics for your library version. **Idempotency — the governing safety rule:** retries and even timeouts create ambiguity. After a *response* timeout the server may have already processed the request. So: - Retry only **idempotent** operations (GET/PUT/DELETE/HEAD). - For writes (POST), require an **idempotency key** so the server de-duplicates, or don't retry. - Treat 'sent-but-no-response' as *maybe-succeeded*, not *definitely-failed*. **Latency budget / deadline propagation:** worst-case time ≈ `maxAttempts × (attemptTimeout) + sum(backoffs)`. This must be **shorter** than the deadline your own callers grant you; otherwise you'll be killed by *their* timeout while still retrying — **deadline inversion**. Propagate deadlines: shrink inner budgets as the remaining deadline shrinks. **Retry amplification across a system:** if service A retries B, and B retries C, retries multiply geometrically down the chain, turning a small blip into a load spike. Mitigate with: low retry counts (2–3), retry budgets (cap the % of traffic that is retries), circuit breakers to cut chains, and jitter. **Scoping & observability:** one breaker per remote dependency (or per critical endpoint) — sharing couples unrelated health. Emit metrics/events: retry counts, breaker state transitions, fallback rate, timeout rate. Alert on OPEN breakers and rising fallback rates. **Config location:** centralize this in a reusable `WebClient` builder customizer / a resilience wrapper so every client gets consistent timeouts, and per-dependency retry/breaker policy — rather than sprinkling ad-hoc operators. **When to simplify:** not every call needs all five layers. A read from a fast internal service might just need a timeout + fallback. Reserve the full stack for critical, flaky, or expensive dependencies.