skip to content

Why does kubeadm certs renew never replace a Kubernetes cluster's CA, and how would you rotate the cluster CA without an outage?

level: seniorimportance: nice to knowfreq 26%

answer

  1. renewal versus trust change
  2. both roots accepted at once
  3. every verifier needs the bundle
  4. kubelets won't rotate soon enough
  5. remove old root last

basics

~20 s

Replacing a Kubernetes cluster CA changes what every component trusts, so kubeadm refuses to do it automatically. Rotating it without an outage means trusting old and new CAs together, re-issuing every leaf from the new CA, then removing the old CA.

solid answer

~40 s

A CA is what every component trusts: kube-apiserver's `--client-ca-file`, kubelets, kubeconfigs, pods reading `kube-root-ca.crt`. Swapping it in one step breaks every client that trusts only the old one, which is why kubeadm's renewal code skips CAs. A safe rotation has three phases. First, distribute a **bundle** of old plus new CA to every place that verifies certificates, so both are trusted. Second, re-issue every leaf and kubeconfig from the new CA, including kube-controller-manager's signing key so new kubelet certificates come from it. Restart components one node at a time and re-bootstrap the kubelets. Third, once nothing still presents an old-CA certificate, remove the old CA from the bundles and restart again. Pods need a restart to pick up the new ServiceAccount root CA.

go deeper

for a junior

Remember that renewing a certificate and replacing its CA are different jobs, and kubeadm only does the first.

for a middle

List every place the cluster CA is trusted and explain why an in-place CA swap breaks them all at once.

for a senior

Sequence the overlap, re-issue and clean-up phases, including forced kubelet re-bootstrap and rolling pods for the new root CA.

for a principal

Balance overlap length against exposure: a planned rotation favours a long overlap, a key compromise favours a short one and a planned outage.

## Why kubeadm won't do it The **cluster CA** (`/etc/kubernetes/pki/ca.crt`) is the trust root. Renewing a leaf keeps everyone's trust unchanged: the new certificate is signed by the same CA. Replacing the CA changes trust itself. Every party that verifies a certificate against the old CA fails until it learns the new one. kubeadm's renewal manager explicitly skips CAs for this reason, and `kubeadm certs check-expiration` shows the CA expiry only for information. You rotate the CA when: - its ten-year lifetime is close to running out; - its private key may have leaked (then *every* identity it signed is suspect); - you move to an organisation-managed or external CA. ## Where the cluster CA is trusted On the 9-node bare-metal cluster behind the IoT telemetry ingest gateway, the cluster CA appears in many places: | Where | What uses it | |---|---| | kube-apiserver `--client-ca-file` | verifying client certificates (kubelets, controllers, users) | | kube-controller-manager `--cluster-signing-cert-file` / key | signing kubelet CSRs | | kube-controller-manager `--root-ca-file` | the `kube-root-ca.crt` ConfigMap published into every namespace | | every kubeconfig (`admin.conf`, `kubelet.conf`, ...) | verifying the API server's serving certificate | | kubelets | verifying API server clients (`authentication.x509.clientCAFile`) | | in-cluster clients | verifying the API server through the ServiceAccount CA bundle | The `front-proxy-ca` (checked through `--requestheader-client-ca-file`) and the `etcd-ca` are separate trust roots and rotate on their own schedules. ## A zero-outage sequence The core idea is an **overlap window**: for a while, both CAs are trusted. 1. **Generate the new CA** and keep the old one. Build a bundle file containing both certificates. 2. **Widen trust.** Point every verifier at the bundle: kube-apiserver's `--client-ca-file`, kube-controller-manager's `--root-ca-file`, the kubelet client CA file and the CA data in every kubeconfig. Restart control-plane static pods one node at a time (keeping etcd quorum) and restart kubelets. Now a certificate from either CA is accepted. 3. **Refresh in-cluster trust.** Once kube-controller-manager publishes the bundle, the `kube-root-ca.crt` ConfigMaps update. Pods pick up the change through projected volumes or on restart, so roll the workloads (for example, the gateway's Deployment in batches small enough to keep the 3,400 rps burst covered). 4. **Switch signing.** Give kube-controller-manager the new CA as its signing key, and re-issue the control-plane leaves and kubeconfig client certificates from the new CA (for example, replace `ca.crt`/`ca.key` with the new pair and run `kubeadm certs renew all` on each control-plane node). Restart components again. 5. **Re-issue kubelet certificates.** Kubelets rotate on their own schedule, which could be months. Force it by re-bootstrapping each kubelet, or by deleting its current certificate and restarting it with a bootstrap credential, so it gets a certificate from the new CA. 6. **Update external holders**: user kubeconfigs, CI systems and anything that pinned the old CA. 7. **Narrow trust.** When nothing presents an old-CA certificate any more, remove the old CA from every bundle and restart once more. ## Things that go wrong - **Skipping the overlap.** Replacing `ca.crt` in place breaks every kubelet and kubeconfig at the same moment. - **Forgetting nodes joining later.** A node joining with `kubeadm join` checks a CA public-key hash, which changes with the CA, so update the join instructions. - **Stale in-cluster clients.** An in-cluster client that read the old CA bundle once and cached it rejects the API server's new serving certificate until the pod restarts. - **A leaked key is worse.** With a compromised key, the overlap window is itself exposure, so shorten it and accept more disruption. ## Interview framing The answer interviewers want: renewal changes identity, rotation changes trust. Trust changes need a period where both old and new are accepted, a controlled re-issue, and a clean-up step. ## Verifying each phase Before narrowing trust, prove that nothing still depends on the old CA: - Run `openssl x509 -noout -issuer` against every control-plane leaf and kubeconfig client certificate, and confirm the issuer is the new CA. - On every node, check the issuer of `kubelet-client-current.pem`. The 6 workers are the ones most often forgotten. - Check that `kube-root-ca.crt` in a sample of namespaces contains the bundle, then the new CA only. - Watch the API server's authentication failures during each restart. A rising count means some client still presents an old-CA certificate, so stop and find it before continuing.

  • The cluster CA's private key has leaked. What changes about the rotation plan?
    Speed beats smoothness. Anything the leaked key signs is trusted for as long as the old CA stays in any bundle, so keep the overlap window short or skip it. Accept a planned outage while every component and kubelet gets new certificates, then check RBAC and audit logs for identities the attacker could have minted.
  • Why don't Kubernetes kubelets pick up a certificate from the new CA as soon as kube-controller-manager signs with it?
    A kubelet requests a new client certificate only when its rotation deadline arrives, at 70-90% of its current certificate's lifetime. That may be months away. Until then it presents the old-CA certificate, so either keep the old CA trusted until they all rotate, or force a re-bootstrap.

Changing the locks on a building while people still work there: you install doors that accept both old and new keys, hand everyone a new key, and only then disable the old ones.

saying these in an interview costs you the question

  • kubeadm certs renew all rotates the CA if you pass the right flag
  • Replacing ca.crt and ca.key in place is enough if you restart everything
  • Rotating the cluster CA also rotates the etcd and front-proxy CAs
  • Kubelets switch to the new CA immediately after the signer changes
  • In-cluster pods never need a restart after a CA change