A reverse proxy is configured to try the next upstream when a request fails, allowing up to two extra attempts with a 10-second read timeout each, while the calling client gives up after 15 seconds. Walk through what the client and the backends actually experience.

## The arithmetic nobody does Retry settings and timeout settings are usually configured by different people at different times, and each looks reasonable alone. Multiplied, they are not: ``` attempts (1 initial + 2 retries) = 3 per-attempt read timeout = 10s proxy worst case = 30s client budget = 15s ``` The first thing to establish is whether the retries can *ever* produce a result the client will see. Here, only if the failing attempts fail fast. A connection refused returns in milliseconds, so three quick failures plus a successful attempt land comfortably. But a retry triggered by a **read timeout** consumes the full ten seconds before it even begins, which means retry number two starts at t=20s — five seconds after the client stopped listening. So: retries on connect failures are useful here, retries on timeouts are not. That distinction is the whole design. ## What the backends experience From the backend fleet's point of view, one user request is up to three executions. Add the client's own retries — if it tries three times too — and a single user action can be nine backend executions. That is fine when the fleet is healthy and catastrophic when it is not, because the condition that triggers retries is precisely the condition where extra load is most harmful. The slow-backend case deserves spelling out. If a backend is not dead but merely saturated, every attempt holds one of its workers for the full ten-second timeout before being abandoned. Retrying *adds* concurrent work to a system whose problem is too much concurrent work. And if one member of the pool is slow, retries redirect its overflow onto the healthy members, which is helpful in small doses and is how a partial degradation becomes a total one in large ones. ## Which failures are safe to retry at a proxy A proxy has less information than the client does. It cannot see application semantics; it sees a TCP connection and some bytes. The conservative classification: - **Never wrote a byte to the upstream** — connection refused, connect timeout, DNS failure. The application demonstrably did not see this request, so another member is safe for any method. - **Wrote the request, no response** — a read timeout or a reset mid-flight. The application may have completed the work. Retrying is only safe for idempotent methods, or for requests carrying an idempotency key the application honours. - **Got a 5xx** — the backend answered, so it processed something. Whether that is retryable is an application-level question the proxy cannot answer for itself. The practical setting is to enable retries for the first class by default, and to enable the second only where the request semantics justify it. nginx expresses these conditions in `proxy_next_upstream`; HAProxy's `option redispatch` covers the reconnect-to-another-server case. The default sets in these products are not always as narrow as you would choose, so read them rather than assuming. ## Making the numbers fit Work backwards from the client: ``` client budget 15s safety margin 3s usable proxy budget 12s attempts 3 per-attempt read timeout 4s ``` Four seconds per attempt is a real change to behaviour — a legitimately slow endpoint will now fail — so this number has to come from the endpoint's latency distribution, not from the retry maths alone. If p99 is eight seconds, three attempts do not fit and the honest answer is fewer attempts, not a shorter timeout that fails healthy requests. Two more guards belong here. First, a cap on the *proportion* of traffic that may be retries, so a fleet-wide failure cannot multiply total load — retry budgets as a discipline are covered under resilience patterns, but the proxy is where you enforce them for this tier. Second, backoff between attempts, which matters less at a proxy than in a client because the proxy is usually moving to a *different* upstream rather than hammering the same one. ## How it shows up in an incident The signature is a backend request rate that rises when the error rate rises, in a fixed ratio close to your attempt count, while the client-observed request rate stays flat. If your dashboards only show requests *received* by the proxy, the multiplication is invisible; instrument attempts, not just requests, or you will diagnose the wrong thing.