How should you decide what value to set a downstream call's timeout to, and what specifically goes wrong if a caller's timeout is set longer than the callee's own internal processing timeout (or its thread-pool queue wait)?

## Choosing the value is a statistical decision Choosing a timeout value is fundamentally a statistical decision, not an arbitrary one: the right timeout for a call to a given downstream endpoint should be derived from that endpoint's actual observed latency distribution, commonly its p99 or p99.9 response time under normal, healthy load, plus a modest safety margin, rather than a round number picked because it 'feels long enough.' The reasoning is direct: | Where the timeout is set | What follows | |---|---| | at, say, the median (p50) response time | you'll routinely kill perfectly healthy requests that simply landed in the slower half of the normal distribution, producing a steady background rate of false-positive failures that have nothing to do with the dependency actually being broken | | at p99 plus margin | you accept that roughly 1% of genuinely healthy requests might still be slow enough to threaten the timeout, but the vast majority of legitimate variance is tolerated, while a request that's an order of magnitude slower than even the worst normal case is reliably caught and failed fast | Teams that skip this and hardcode an intuitive round number, 'let's just say 5 seconds', usually end up either: - **far too generous** — masking real problems, allowing threads to block for much longer than the dependency ever legitimately takes, delaying detection of an actual outage; - or, less often but just as harmful, **too tight for a legitimately bursty endpoint** — generating alert fatigue from constant false timeouts that erode trust in the alerting until real incidents get ignored too. ## When the caller waits longer than the callee ever would The specific failure mode of a caller's timeout being longer than the callee's own internal timeout or queueing tolerance is a particularly insidious one because it's invisible from the caller's side until you look closely. Imagine a callee service that itself enforces, say, a 2-second internal processing timeout on any single request (perhaps it queries a database with a 2s query timeout), but the calling service's HTTP client is configured with a 10-second read timeout. When the callee is healthy, this mismatch does nothing visible, most calls finish well under 2s and the caller's generous timeout never matters. But under degraded conditions, say the callee's thread pool starts backing up because of a slow database, new inbound requests to the callee sit in its own request queue before a worker thread even picks them up. If the callee has no queue-wait timeout of its own (only a processing timeout that starts once a thread picks up the work), a request can sit queued for many seconds, and the calling service's overly generous 10-second timeout does nothing to stop it from continuing to wait, worse, it does nothing to stop the caller from sending more concurrent requests into that same backed-up queue, because from the caller's perspective nothing has failed yet. The caller's threads pile up waiting on the callee's queue, and the callee's queue keeps growing because the caller isn't failing fast enough to signal back-pressure. This is exactly the mechanism behind many real 'slow death' cascading failures: a caller with too-generous a timeout doesn't merely tolerate a slow callee, it actively participates in making the callee slower, by not applying back-pressure early enough. ## The fix The fix has two parts. 1. **First, on the caller.** The caller's timeout for a given call should generally be set at or below what the callee's own worst reasonable processing time is, informed by the callee's published SLA or observed p99 latency, so that the caller never usefully waits longer than the callee would ever legitimately take to answer. 2. **Second, and just as important, on the callee.** The callee itself should enforce its own internal bounds, a request-queue-wait timeout in addition to a processing timeout, so it fails fast internally on requests it can't get to promptly, returning a fast, explicit error (which the caller can act on, e.g., fail over elsewhere or open a circuit breaker) instead of silently holding the request in an unbounded queue. ## Where this bug lives in real fleets Systems like Envoy expose both a per-route timeout and separate idle-timeout or queue-related settings for exactly this reason, and Java thread-pool-backed servers (Tomcat, older Spring MVC deployments) have historically been a common source of this exact bug, because a bounded worker thread pool with an unbounded request queue in front of it lets requests wait indefinitely for a free thread with no timeout ever firing on the queueing itself, only on the processing that starts once a thread is finally free. A concrete real-world instance of this class of incident is well documented in postmortems from companies running large Java service fleets on Tomcat and Jetty during traffic spikes, where callers configured with generous multi-second timeouts kept sending traffic into services whose request queues had already grown into the tens of seconds, amplifying rather than containing the outage.