In an Istio mesh, where does a workload's mutual-TLS certificate come from, what identity does that certificate carry, and how is it kept fresh?

## Why this matters before any policy does Every identity-based rule you can write in the mesh — a principal in an authorization policy, a peer shown in telemetry — is only as meaningful as the certificate underneath it. Understanding the issuance flow tells you what an identity actually asserts and, more importantly, what it does not. ## The issuance flow 1. **Key generation, in the pod.** The istio-agent process running alongside the proxy generates a private key inside the pod's memory. Nothing external ever sees it. 2. **A certificate signing request goes to istiod.** Istiod embeds the mesh certificate authority. The agent authenticates the request using the pod's projected service-account token, mounted by the platform with an audience scoped for this purpose. 3. **Istiod validates the token** against the Kubernetes API — confirming that this token really belongs to a live service account — and only then signs. 4. **The signed certificate comes back** and the agent serves it to the proxy over a local secret discovery service on a Unix domain socket. The proxy loads it into its TLS transport configuration and starts using it for both inbound and outbound mesh connections. Two properties fall out of this design that are worth stating explicitly. The private key **never crosses the pod boundary**, so there is no Secret object holding workload keys to leak, and a compromise of one pod does not yield another pod's key. And issuance is **online and continuous**, not a bootstrap step — a workload that cannot reach istiod will keep serving with its current certificate until it expires, then start failing. ## What the identity actually says The identity is encoded as a URI subject alternative name in the SPIFFE format: ``` spiffe://cluster.local/ns/payments/sa/ledger ``` `cluster.local` is the mesh's trust domain, configurable at install time and important once you federate meshes. The remainder is the namespace and the service account. In authorization rules the same identity is written **without the scheme**: ```yaml from: - source: principals: ["cluster.local/ns/payments/sa/ledger"] ``` The consequence that catches teams out: **the granularity of mesh identity is exactly the granularity of your service accounts.** If five Deployments in a namespace all run under the namespace's default service account, they share one identity and are completely indistinguishable to any policy you write. Introducing identity-based authorization to an existing cluster therefore usually begins with a boring prerequisite — give every workload its own service account — long before anyone writes a policy. Equally, the identity says nothing about *which Service* was addressed; it identifies the caller, not the route. ## Rotation Workload certificates are deliberately short-lived — a 24-hour lifetime is the default — and the agent renews well before expiry, conventionally around half of the remaining lifetime, so that a temporary control-plane outage has hours of headroom rather than minutes. Because the swap happens through the secret discovery API, the proxy picks up the new material without restarting and without dropping connections, and the application is never involved. Short lifetimes are the mesh's substitute for revocation: rather than distributing revocation information, an identity that should no longer exist simply stops being reissued. The root of trust is separate. By default istiod generates a self-signed root and holds it, which is convenient and appropriate for a single cluster. For anything beyond that — multi-cluster meshes, or an organisation that wants mesh identities chained to its own hierarchy — you plug in an intermediate by creating a secret named `cacerts` in the istiod namespace containing the certificate, key, root certificate and chain. Rotating that root is a genuine operational exercise, because every workload's chain must remain verifiable throughout. ## Diagnosing it When mesh TLS is failing, the questions in order are: can the agent reach istiod, is the certificate present and unexpired in the proxy's secret configuration, and does the identity in it match the service account you expected. The last one is where most surprises live — a workload running under an inherited default service account is the usual reason a carefully written principal rule never matches.