An Envoy route is configured with `timeout: 3s` and a `retry_policy` of `num_retries: 3` with `per_try_timeout: 3s`, but slow upstreams are never retried in production. Why not, and how should the two timeout values relate?

## Two different clocks Envoy exposes two timeouts on the same route, and they measure different things. - **`timeout`** (route-level, default 15 seconds) is the overall request timeout. It bounds the entire request as seen by the client, *including every retry attempt and the backoff between them*. Setting it to `0s` disables it. - **`per_try_timeout`** (inside `retry_policy`) bounds a **single attempt**. If it is not set, each attempt is bounded only by the global route timeout. Because the overall timeout is a hard ceiling on the whole sequence, `per_try_timeout >= timeout` makes the retry policy unreachable: the first attempt uses the full budget, the overall timeout fires, Envoy sends a 504 with response flag `UT`, and no further attempt is made. The configuration looks like it retries; the stat `cluster.<name>.upstream_rq_retry` stays at zero, which is how you confirm it. ```yaml route: cluster: payments timeout: 3s # covers ALL attempts retry_policy: retry_on: "5xx,reset,connect-failure" num_retries: 1 per_try_timeout: 1.2s # must be a fraction of 3s ``` ## Sizing the two numbers A workable rule: pick the overall budget from what the caller can actually wait for, decide how many attempts that budget can afford, then divide, leaving headroom for backoff. With a 3s budget and two total attempts, about 1.2–1.4s per try is right. More attempts inside the same budget means each attempt gets less time, which makes the retry more likely to be cut short than to succeed — that trade is the real design decision, not the value itself. Envoy's default retry backoff uses an exponential schedule with a base interval of 25ms and a maximum of 250ms; `retry_back_off` lets you set `base_interval` and `max_interval` explicitly. The backoff time is spent inside the overall budget too. ## retry_on must actually match the failure Retries only happen for conditions listed in `retry_on`. The common values are `5xx`, `gateway-error` (502/503/504), `reset`, `connect-failure`, `retriable-4xx`, `refused-stream`, `retriable-status-codes` and `envoy-ratelimited`. A per-try timeout is treated as a 5xx-class failure, so `retry_on: 5xx` covers it while `connect-failure` alone does not. A second common cause of "retries never happen" is simply that the failure mode in production is not in the `retry_on` list. ## The other silent skips Even with the timeouts sized correctly, Envoy may decline to retry: - **Concurrency limits.** Retries are gated by the cluster's `circuit_breakers` `max_retries` threshold (default 3 concurrent retries) or, if configured, `retry_budget` with `budget_percent` and `min_retry_concurrency`. Overflow increments `cluster.<name>.upstream_rq_retry_overflow` and the original failure is returned as-is. - **Body buffering.** A request whose body exceeds the buffer limit cannot be replayed, so it will not be retried. - **Response already started.** Once upstream response headers have been forwarded downstream, the attempt is committed. ## Per-request overrides Envoy honours request headers that override route settings when the route allows it: `x-envoy-retry-on`, `x-envoy-max-retries`, `x-envoy-upstream-rq-timeout-ms` and `x-envoy-upstream-rq-per-try-timeout-ms`. These are useful for a specific slow client path, and they are a hazard at the edge — a proxy facing untrusted clients should strip them rather than let callers dial up their own retry count against your backends. ## Where retries make things worse Retrying a *slow* upstream, as opposed to a failed one, adds load to something already struggling. Two guards matter. First, cap total attempts low — one retry is usually the right answer for an interactive path. Second, prefer a different host on the retry: `retry_host_predicate` with `envoy.retry_host_predicates.previous_hosts` plus `host_selection_retry_max_attempts` makes Envoy avoid the endpoint that just failed, which turns a retry into a real second chance rather than a second hit on the same sick host. ## Confirming the fix After changing values, watch three counters per cluster: `upstream_rq_retry` (attempts made), `upstream_rq_retry_success` (attempts that rescued a request) and `upstream_rq_retry_overflow` (attempts refused by the limit). A retry policy where success stays near zero is buying latency and load for nothing and should be turned down or off.