A platform team running Istio at 3,000-pod scale sees intermittent 503s during rolling deploys and a slow but steady rise in per-node memory pressure over months. Walk through the sidecar-specific failure modes that could explain each symptom.

## One structural fact behind both symptoms Both symptoms trace back to the same structural fact about sidecars: they **multiply per-pod**, and they have their own **independent lifecycle** from the application container, so any assumption that 'the app terminating cleanly' or 'the app's resource footprint' fully describes the pod's behavior is wrong once a sidecar is in the picture. ## The rolling-deploy 503s Take the rolling-deploy 503s first. When Kubernetes terminates a pod during a rollout, it sends SIGTERM to every container roughly simultaneously and starts a grace-period countdown, while in parallel the pod is removed from the relevant Service's endpoint list so new traffic stops being routed to it. These two things - 'stop being sent new traffic' and 'container processes receive SIGTERM' - are **not perfectly synchronized**; there's a well-known race where a pod can still receive a request from a caller whose local sidecar hasn't yet learned the endpoint disappeared, at the exact moment the pod's own sidecar or app container has already begun shutting down. If the application container exits (or stops accepting connections) before the sidecar has finished proxying the last few in-flight requests, or if the sidecar itself exits first and stops accepting new connections while the app is still theoretically reachable, the caller sees a connection refused or an abrupt reset, which upstream often surfaces as a 503. Before Kubernetes' native sidecar container ordering (`restartPolicy: Always` init containers), there was no guarantee the sidecar would outlive the app container during shutdown; the standard mitigations were: - a **preStop hook** that sleeps for a few seconds before sending SIGTERM, giving endpoint-list propagation time to catch up across the mesh - configuring the sidecar to **drain existing connections gracefully** rather than dropping them immediately Getting this wrong at 3,000-pod scale is exactly when it becomes visible - rollouts happen constantly, so even a small window of unlucky timing per pod produces a steady trickle of user-visible errors. ## The memory-pressure trend The memory-pressure trend is a more structural, harder-to-fix cost: it's the direct **multiplicative overhead** of the sidecar pattern made concrete. Every pod's Envoy process holds: - its own **configuration cache** (the routing/cluster/endpoint tables pushed by the control plane) - its own **connection pools** to every backend it talks to - its own **stats/metrics buffers** None of that is shared across pods, because sidecars are deliberately per-pod, not per-node. As the cluster grows to 3,000 pods and the mesh's service graph grows (more services, more possible destinations each proxy needs to know about), each individual sidecar's config payload and connection-pool footprint tends to grow too, not just stay flat - a proxy serving a service that talks to 50 downstreams needs meaningfully more cached config and more open connections than one talking to 3. This is why Istio has features like **Sidecar resource scoping** (limiting which parts of the mesh's configuration a given sidecar actually needs to know about, rather than every proxy holding the entire mesh's config) specifically to control this growth; without that tuning, memory scales roughly with both pod count and mesh size, and teams that provisioned node capacity based on application memory needs alone find themselves surprised months later when sidecar overhead has quietly become a large fraction of total cluster memory. ## The principle both symptoms illustrate The broader principle both symptoms illustrate is that a sidecar is **not a free abstraction** layered invisibly over the app - it has its own lifecycle that must be explicitly coordinated with the app's (hence the shutdown-ordering fix), and its own resource curve that scales with fleet and mesh topology, not with the application's own workload (hence the memory trend). Diagnosing either requires looking past application-level metrics and dashboards entirely and into **sidecar-specific telemetry**: - the proxy's own admission/shutdown and access logs for the 503s - per-container (not just per-pod) memory metrics broken out by container name for the memory trend, since a pod-level memory graph that only shows the sum would hide which container is actually growing ## A concrete reference A concrete real-world reference is Istio's documented graceful pod shutdown guidance and its introduction of native Kubernetes sidecar support specifically to close the termination-ordering race described above, alongside Sidecar resource-scoping as the documented mechanism for controlling exactly the kind of mesh-wide config/memory growth described here.