In Istio's architecture, what specifically does the control plane (istiod) do versus what the data plane (the Envoy sidecars) does, and how does configuration get from one to the other?

## The core architectural idea The control plane / data plane split is the core architectural idea that makes a service mesh operable at scale, borrowing directly from how **SDN** (software-defined networking) separates route computation from packet forwarding. ## The data plane The data plane is the aggregate of every sidecar proxy running in the mesh — in Istio, that's Envoy instances, one per pod. Each Envoy does the actual, per-request work: - terminating and originating TLS; - matching request routes against configured rules; - picking an upstream endpoint via a load-balancing policy; - applying timeouts and retry budgets; - enforcing circuit-breaking connection/request limits; - and emitting access logs and metrics. Crucially, none of this is centralized — every Envoy makes its own per-request decisions locally using configuration it already has cached in memory. This is what lets the data plane survive control plane outages: if istiod goes down, existing Envoys keep routing traffic exactly as last configured; the mesh doesn't depend on the control plane for every single request. ## The control plane The control plane, istiod, is the single binary Istio consolidated its earlier multi-component design (Pilot, Citadel, Galley) into. It does three main jobs. 1. **First, discovery and translation:** it watches the Kubernetes API server for Services and Endpoints, plus Istio's own CRDs — `VirtualService` (routing rules like traffic splitting or header-based routing), `DestinationRule` (load-balancing policy, connection pool limits, outlier detection for circuit breaking), and `PeerAuthentication/AuthorizationPolicy` (mTLS mode and access control) — and compiles all of that into Envoy's native configuration format. 2. **Second, distribution:** it pushes that compiled config down to every connected sidecar over a long-lived gRPC stream using the xDS protocol family. 3. **Third, certificate authority:** istiod runs Istio's built-in CA, issuing short-lived X.509 certificates to each workload's Envoy for mTLS, and rotating them automatically before expiry — this is what lets the data plane do mutual TLS without any app-level cert management. | Discovery service | What it carries | |---|---| | `CDS` | clusters, i.e. upstream service groups | | `EDS` | endpoints, i.e. actual pod IPs | | `LDS` | listeners, i.e. what ports/protocols a proxy exposes | | `RDS` | routes, i.e. routing rules | | `SDS` | secrets, i.e. TLS certs/keys | ## Why the split exists Why the split exists: it lets you change routing or security policy across an entire fleet by editing a handful of YAML objects instead of touching every service, and it keeps the actual traffic path fast and simple — Envoy is optimized purely for high-throughput proxying, while istiod does the comparatively rare, heavier work of watching API state and compiling config. It also isolates blast radius: a bug or outage in the control plane degrades the mesh's ability to *adapt* (new pods take longer to get their config, policy changes stall) without instantly breaking traffic already flowing, because data plane proxies are eventually-consistent, cached consumers of control-plane state rather than synchronous dependents. ## The trade-offs Trade-offs: this design trades immediate consistency for resilience. - **Immediate consistency.** When you apply a new `VirtualService`, propagation to every sidecar isn't instantaneous — a canary rollout or security policy tightening isn't atomic across the fleet, and for a short window some proxies are on old config and some on new. - **There's also a genuine operational cost to the control plane itself.** istiod needs its own capacity planning, monitoring, and upgrade path, and a misbehaving control plane pushing malformed config can degrade every sidecar in the mesh simultaneously — a fleet-wide blast radius risk that didn't exist before the mesh. ## What goes wrong Failure modes in production: - **The most reported is xDS push starvation.** istiod under memory or CPU pressure (common in clusters with high pod churn) falls behind pushing config, so newly started pods sit with an empty or stale config snapshot and route traffic incorrectly or reject it. - **Another is CA unavailability.** If istiod's cert-issuance path is disrupted, new workloads can't get a certificate and fail mTLS handshakes, while already-running workloads keep working until their certs expire (typically 24 hours by default), giving operators a runway to fix it before an outage cascades. - **A third is config drift debugging.** Because each Envoy applies config independently, operators sometimes see inconsistent behavior across replicas of the same service during a rollout, and diagnosing it requires comparing per-pod proxy config dumps rather than assuming one global source of truth is instantly authoritative everywhere. ## Where it shows up A concrete real-world instance of this exact split: Istio's own architectural history is the clearest example — pre-1.5 Istio ran Pilot, Citadel, Galley, and a Sidecar Injector as separate control-plane binaries; they were merged into the single istiod process specifically to reduce control-plane operational overhead while keeping the underlying control/data plane separation intact.