Sidecar and Ambassador

## What the pattern is The sidecar pattern deploys a **helper container** alongside a **primary application container** within the same deployment unit - most commonly a Kubernetes pod - so that the two containers share a **network namespace** (same localhost, same IP) and can share a filesystem via mounted volumes. - Because they share the network namespace, the app can talk to the sidecar (and vice versa) over `127.0.0.1` with no DNS lookup, no extra hop, and no need to open the connection to the outside world. - Because they can share volumes, the sidecar can tail a log file the app writes to disk, or write a freshly rotated certificate that the app picks up on its next connection. Crucially, sidecar and app are **scheduled, scaled, and terminated together as a single atomic unit**: if Kubernetes moves the pod to a new node, both containers move together; if you scale the deployment from 3 to 30 replicas, you get 30 sidecars too, one per app instance, with no separate scaling knob. ## The problem it solves The problem the pattern solves is that many concerns are needed by every service in a fleet: - request logging - metrics emission - mutual TLS termination - config hot-reload - service-discovery lookups - retries and circuit breaking But reimplementing them inside every service's codebase in every language creates massive **duplication** and **version-skew risk** (the Java team's retry library diverges from the Go team's). Rather than writing an in-process library that each service must import, compile against, and independently upgrade, you package the concern once as a standalone container image and attach it to any pod, regardless of what language or runtime the app inside is written in. Upgrading the cross-cutting behavior becomes a matter of bumping the sidecar image tag and rolling pods, not touching application code or triggering a rebuild in every language's dependency graph. ## Where it shows up - **TLS/mTLS** is the most common concrete use: instead of every service linking a TLS library and managing certificate rotation itself, the app talks plaintext to its sidecar over localhost, and the sidecar wraps outbound traffic in mTLS and unwraps inbound traffic, handling certificate issuance and rotation from a central authority behind the scenes. - **Log/metrics shipping** is another common one: the app writes plain text lines to stdout or a mounted volume, and a sidecar (e.g., a log-forwarder container) tails that stream and ships it to a central aggregator, so the app never needs a logging-backend SDK at all. ## The trade-off The trade-off is **resource and operational cost multiplied by replica count**. Every pod now runs (at minimum) two processes, so CPU/memory requests, container start time, and the surface area for things to go wrong all roughly double per instance - at 500 replicas you now manage 500 extra sidecar processes, each needing its own resource limits, health checks, and image lifecycle. There is also a small **latency** cost: traffic that used to go directly out of the process now traverses an extra localhost hop through the sidecar's proxy logic (typically sub-millisecond, but not zero, and it shows up under tail-latency scrutiny at very high QPS). ## Failure modes Failure modes in production center on lifecycle ordering and resource starvation. 1. **If the sidecar isn't ready before the app starts sending traffic** (e.g., an mTLS-terminating sidecar hasn't finished loading certificates yet), early requests fail with connection-refused or TLS-handshake errors - this is why Kubernetes added native sidecar container support (`restartPolicy: Always` init containers) so the kubelet starts sidecars before the main container and keeps them running throughout the pod's life. 2. **The mirror problem happens at shutdown**: if the sidecar exits before the app finishes flushing its last few requests or log lines (a common issue before native sidecar ordering existed, where sidecars were plain containers with no guaranteed shutdown order), those final requests are silently dropped. 3. **Resource contention** is the other classic failure: if the sidecar isn't given its own CPU/memory limits, a traffic spike can starve it, and because the app is entirely dependent on it (e.g., all egress goes through it), the app effectively goes down even though its own container is healthy. ## A concrete example A concrete real-world example is Istio's **Envoy sidecar** (`istio-proxy`), injected automatically into every pod in a mesh via a mutating admission webhook; it intercepts all inbound and outbound traffic via iptables rules, handling mTLS, retries, and telemetry uniformly across services written in a dozen different languages, without any of those services linking an Istio-specific library.