When a platform team configures retry and timeout policy in a service mesh's data plane (e.g. Istio's VirtualService/DestinationRule) instead of in each service's application code, what do they gain, what do they give up, and where can this go wrong?

## What traffic management covers Traffic management in a sidecar mesh covers two related but distinct capabilities: - **Routing** — deciding which version or subset of a service a request goes to: traffic splitting for canaries, header-based routing for A/B tests, mirroring traffic for shadow testing. - **Resilience** — retries, timeouts, circuit breaking/outlier detection, applied to that routed traffic. Both get expressed as declarative configuration attached to the mesh rather than written as code inside each service, and both get enforced by the sidecar at the moment it forwards a request. ## How it is configured Mechanism: in Istio, a `VirtualService` defines routing rules — for example, sending 95% of traffic to one service version and 5% to another for a canary rollout, or routing requests carrying a specific header to a debug deployment — plus per-route resilience settings like retries (attempts, per-try timeout, and which conditions count as retryable) and an overall request timeout. A `DestinationRule` separately configures connection-pool limits and outlier detection, ejecting an endpoint from the load-balancing pool after it returns a run of consecutive errors. All of this compiles down to sidecar configuration and is enforced entirely inside it: the application just makes one HTTP call as normal, and if that call fails, the sidecar, not the app, decides whether to retry, how long to wait, and when to give up. ## Where it came from Why this exists: before meshes, retry/timeout logic lived inside each service, usually via a resilience library that every team had to independently adopt, configure sensibly, and keep updated — inconsistency was the norm, with some services retrying aggressively, others not retrying at all, and timeouts set arbitrarily differently across a call graph. Centralizing this in the mesh's data plane means a platform team can: - set and audit resilience policy for the entire fleet from one place; - get it applied uniformly regardless of what language a service is written in; - change it (e.g. tighten a timeout causing cascading slowness) without asking every team to redeploy. ## The idempotency blind spot Trade-off, precisely stated: the mesh sidecar operates at the HTTP/gRPC transport layer and only sees generic request/response semantics — **method, path, headers, status code**. It has no idea whether a particular POST is idempotent and safe to retry blindly, or has side effects where retrying could double-charge a customer or duplicate a write, because that's business-logic knowledge the sidecar structurally cannot have. Application-level resilience code can be written with exactly that context — a service's own code can know a specific endpoint is idempotent because it carries an idempotency key and retry accordingly, in a way a generic mesh policy can't know to distinguish from a non-idempotent endpoint on a different route. So mesh-level retries are best applied to genuinely safe, generic cases and layered with per-route granularity rather than one blanket policy for a whole service. ## What goes wrong - **The most dangerous failure mode this trade-off produces is the retry storm.** If a downstream service starts failing because it's overloaded, and every upstream caller's sidecar is configured to retry those failures two or three times, the mesh can multiply the effective request rate hitting the already-struggling service at exactly the moment it can least handle it, turning a partial degradation into a full outage. This is why mature mesh configurations pair retries with tight retry budgets (capping total retry volume, not just per-request attempts), short per-try timeouts, and outlier detection that ejects a consistently-failing endpoint from the load-balancing pool rather than continuing to hammer it. - **A second failure mode is a mismatch between the mesh's timeout and the application's own internal timeout.** If the mesh's request timeout is shorter than a downstream operation the application is mid-way through, the client gets an error while the server-side operation keeps running to completion unaware it was already abandoned, occasionally causing subtle data or resource issues. - **A third is simply operational confusion.** An engineer debugging elevated latency looks exclusively at application code and logs, not realizing the actual retries are happening invisibly in the sidecar and inflating the effective latency and load their service is generating. ## Where it shows up A concrete real-world pattern: Istio's documented canary deployment workflow uses exactly this weighted-routing mechanism — shifting traffic percentage from an old to a new service version incrementally while watching error-rate and latency metrics at each step, entirely through mesh config changes and with zero code changes or redeploys of the services themselves.