An Istio VirtualService for a slow service sets `timeout: 10s` together with `retries: {attempts: 3, perTryTimeout: 5s}`. Explain how those two settings interact, and why a chain of services each configured this way can turn a slowdown into an outage.

## Two different clocks The VirtualService exposes both settings on the same HTTP route entry, and they measure different things: ```yaml - route: - destination: {host: payments, subset: v1} timeout: 10s retries: attempts: 3 perTryTimeout: 5s retryOn: connect-failure,refused-stream,unavailable ``` - **`timeout`** is the deadline for the request as the caller experiences it: the first attempt, every retry, and everything in between. When it fires, the caller gets a failure regardless of what any attempt was doing. - **`perTryTimeout`** is the deadline for one attempt. When it fires, that attempt is abandoned and — if attempts remain and the outcome is retryable — another is started. The overall timeout always wins. `3 × 5s` wants fifteen seconds; the ten-second budget truncates it. The practical consequence is that the configuration promises three attempts and delivers roughly two, while the upstream has been made to do work for a request whose answer will be discarded. Whenever you set both, do the arithmetic: `attempts × perTryTimeout` should fit inside `timeout` with room to spare, or you should accept explicitly that the last attempt is decorative. ## Retries you did not ask for Istio applies a **default retry policy** to HTTP routes even when the VirtualService contains no `retries` block — a small number of attempts on connection-level failures and a limited set of retryable conditions. This surprises people: a team reasons that they never enabled retries, so a doubled request count cannot be theirs. It can be. Make the policy explicit on any route where the behaviour matters, and use `attempts: 0` to switch it off rather than relying on omission. ```yaml retries: attempts: 0 # explicitly no retries for a non-idempotent path ``` ## Why chains amplify Retry amplification is multiplicative, not additive. Take three hops, each retrying twice after the initial attempt — three total attempts per hop: ``` edge → A : up to 3 requests to A A → B : each of those, up to 3 → 9 requests to B B → C : each of those, up to 3 → 27 requests to C ``` One user request can become twenty-seven at the deepest service. And the amplification is **triggered by** the deepest service being slow or erroring, so the extra load arrives precisely when it is least survivable. This is the classic retry storm: a brief blip at the bottom of the stack is converted by the middle tiers into sustained overload, and the system does not recover on its own even after the original cause clears, because the retries themselves now supply the load. A layered timeout budget is the structural defence. Each hop's `timeout` should be meaningfully **shorter** than its caller's, so an inner call is abandoned before the outer deadline expires. When the outer timeout is shorter than the inner one, the caller gives up while the inner attempt is still running, the work continues to consume capacity, and the retry it triggers piles on top. ## Choosing retryOn honestly `retryOn` takes a comma-separated list of conditions — `connect-failure`, `refused-stream`, `unavailable`, `gateway-error`, `retriable-status-codes`, `5xx` among them. The safe end of that list is the conditions where the request demonstrably **never reached** the application: a connection that failed to establish, or a stream the server refused. Retrying those risks nothing. `5xx` is at the other end: the request may well have been processed and had its side effect before the error was produced, so retrying can duplicate work. Retry-safety is a property of the endpoint, so the setting belongs per route, not mesh-wide. ## Where the neighbouring controls live The `retries` block has no budget or ceiling of its own. What limits concurrent pressure on an upstream is the **DestinationRule's** `trafficPolicy.connectionPool` — `tcp.maxConnections`, `http.http1MaxPendingRequests`, `http.http2MaxRequests`, `http.maxRequestsPerConnection` — and its `outlierDetection`, which takes a persistently failing endpoint out of rotation. A complete answer places the retry policy in the VirtualService and the pressure limits in the DestinationRule, and notes that retries without those limits are a load multiplier with no governor. ## What to say in an interview Start with the two clocks and the arithmetic, name the default retries as something you have to switch off deliberately, then move straight to amplification along the chain and the decreasing-budget rule. Finish on retry safety: attempts are only free when the request never landed.