How do you rotate broker TLS keys/certificates and CA trust without downtime?

## Two distinct things rotate 1. **Leaf certificate + private key** (a single broker's identity in its keystore). Rotated frequently (expiry, key compromise, hostname change). 2. **CA certificate** (the trust anchor in every truststore). Rotated rarely but high-stakes, because *everyone* trusts it. They demand different procedures. ## Leaf rotation (low risk, can be zero-downtime) Kafka can reload SSL stores dynamically: - **File-watch reload:** the file-based store loader tracks the keystore/truststore file's modification time and reloads when it changes — so atomically replacing the keystore file (write new, then rename over) makes the broker start presenting the new cert on subsequent handshakes, no restart. - **Dynamic broker config (KIP-226):** `ssl.keystore.location`/`ssl.keystore.password`/`ssl.key.password` (and the type/key/cert for PEM) are **per-listener dynamically updatable** via `kafka-configs.sh --alter --entity-type brokers`. Updating them triggers a reload of that listener's `SslEngineFactory`. Existing TLS connections keep their already-negotiated session; **new** connections use the new cert. Because Kafka clients reconnect/re-handshake over time (and on metadata refresh), the fleet converges without dropping traffic. To force faster convergence you can do a rolling reload broker-by-broker. ## CA rotation (high risk — must overlap trust) The invariant: **never have a peer present a certificate the other side doesn't yet trust, and never remove a CA that some live cert still chains to.** So stage it: **Phase 1 — distribute the new trust anchor.** Add the **new CA** certificate to *every* truststore (all brokers and all clients) alongside the old CA. Now both old- and new-CA leaf certs are trusted. Nothing presents the new CA yet. **Phase 2 — migrate leaf certs.** Roll each broker's keystore to a leaf cert signed by the **new CA** (using the dynamic reload above). Update client certs too if mTLS is in play. At all times every presented cert chains to a CA in everyone's truststore (old or new). **Phase 3 — retire the old CA.** Once *all* leaves are signed by the new CA and verified, remove the **old CA** from every truststore. Do this last, and only after confirming no live cert chains to it. If you reverse the order — e.g. issue new-CA leaves before truststores trust the new CA, or drop the old CA while old-CA leaves are still live — handshakes fail and connections drop. ## Cross-signing / intermediate strategy For large fleets, a **cross-signed** new CA (signed by the old CA) lets new-CA leaf certs validate under the existing trust anchor during transition, shrinking the truststore-distribution window. Equivalently, run a two-tier PKI (root + intermediates) and rotate intermediates while the root stays stable, so truststores rarely change. ## Inter-broker dimension Brokers are clients to each other for replication/controller traffic, so inter-broker connections must also tolerate the overlap: both old and new CAs trusted, leaf rotation rolled. The same phased order applies to the inter-broker listener. ## Automation & hygiene - Use **short-lived certs** issued by cert-manager / Vault PKI emitting **PEM** (KIP-651), so rotation is continuous and boring rather than a rare fire drill. - Monitor cert **expiry** and alert well ahead. - Test the rotation runbook in staging; the failure modes are connection storms, not data loss, but they're user-visible. ## Quick checklist - Leaf: swap keystore file / dynamic-config update → file-watch/SslEngineFactory reload → new connections use new cert. - CA: add new CA everywhere → migrate leaves → remove old CA last.