skip to content

What are the security implications of migrating a live cluster from ZooKeeper to KRaft, and how do you keep the dual-write/bridge period secure?

level: principalimportance: should knowfreq 25%

answer

  1. KIP-866 bridge = dual-write, both ZK + KRaft live
  2. zookeeper.metadata.migration.enable=true
  3. Two trust roots at once → harden BOTH
  4. Principal/authorizer/super.users must match across
  5. Don't tear down ZK or relax it until finalize

basics

~20 s

During ZK→KRaft migration (KIP-866) both systems run together and metadata is dual-written, so you must keep ZooKeeper hardened (SASL/TLS/ACLs) AND secure the new controller listener at the same time. The migration controllers still connect to ZK, so neither attack surface can be relaxed until ZK is removed.

solid answer

~50 s

The KIP-866 migration runs a bridge period: KRaft controllers come up in migration mode, connect to the existing ZooKeeper, and dual-write metadata so brokers can be rolled to KRaft incrementally. Security-wise this means you temporarily have BOTH attack surfaces live: ZooKeeper (with its SCRAM/ACL znodes) and the new controller quorum listener. You must keep ZK fully hardened (SASL, zookeeper.set.acl, TLS, isolation) throughout, since metadata still lives there, while simultaneously securing the controller listener (mTLS/SASL_SSL) and giving migration controllers ZK credentials. Pitfalls: the migration controllers need valid ZK auth so don't break it mid-flight; SCRAM/ACL records must translate cleanly into the metadata log; super.users/principals must be consistent across both. Only after the cluster finalizes KRaft and ZooKeeper is decommissioned can you retire the ZK hardening — and that's the security win: one fewer system to protect.

go deeper

for a junior

Know migration off ZooKeeper exists and that during it both systems run together.

for a middle

Describe the dual-write bridge and that both ZK and the controller listener must be secured at once.

for a senior

Detail the migration-mode config, ZK credential needs, and ACL/SCRAM translation pitfalls.

for a principal

Plan the whole cutover keeping two trust roots hardened, ensure authorizer/principal parity, manage rollback safely, and articulate the end-state surface reduction.

## The migration model (KIP-866) Migrating a *running* cluster from ZooKeeper to KRaft isn't a flip — it's a staged **bridge** process so you don't lose metadata or take an outage: 1. **Provision KRaft controllers in migration mode** (`zookeeper.metadata.migration.enable=true`), and give them ZooKeeper connection + credentials. They form the KRaft quorum but also talk to ZK. 2. **Migrate metadata**: the controllers copy existing ZK metadata into the `__cluster_metadata` log, then enter **dual-write** — every metadata change is written to *both* ZK and the KRaft log so old (ZK-mode) and new brokers stay consistent. 3. **Roll brokers** from ZK-mode to KRaft-mode one by one. 4. **Finalize**: once all brokers are KRaft and you're confident, take controllers out of migration mode; **decommission ZooKeeper**. ## Why this is a security-sensitive window During the bridge you have **two trust roots live at once**: - **ZooKeeper** still holds (and is being written to with) SCRAM creds, ACLs, topic metadata. It must remain hardened the entire time — relaxing `zookeeper.set.acl`, SASL, TLS, or network isolation here would expose the still-authoritative metadata. - **The KRaft controller listener** is now also live and carrying the same metadata. It must be secured (mTLS/SASL_SSL, isolation) from the start of migration, not as an afterthought. So the migration *temporarily increases* attack surface. The end-state reduces it (ZK gone), which is a core security motivation for migrating — but the transition must not be the weak link. ## Concrete pitfalls - **Migration controllers need ZK credentials**: they connect to ZooKeeper, so their JAAS/TLS config must be valid. Misconfiguring it stalls migration; over-broad ZK creds widen exposure. - **Principal/authorizer consistency**: ACLs translate from the ZK-based authorizer to KRaft's StandardAuthorizer. `super.users`, principal naming (SSL DN vs SASL name), and authorizer class must line up, or you can lock out brokers/controllers mid-migration. - **SCRAM translation**: existing SCRAM credentials in ZK must appear correctly in the metadata log; verify clients can still authenticate after the cutover. - **Listener consistency**: brokers gain a controller listener config during migration; ensure it's secured and not accidentally PLAINTEXT. - **Rollback window**: KIP-866 supports reverting to ZK before finalization; keep ZK hardened so a rollback doesn't land on a weakened ensemble. - **No premature ZK teardown**: do not remove ZK ACLs/credentials or shut nodes until finalize is complete and verified. ## End-state security benefit Once finalized: one consensus system, one listener to secure, no ZooKeeper port, no separate ZK JAAS/TLS to maintain, and metadata access fully gated by Kafka's own authN/authZ on the controller listener. The migration's whole security thesis is *shrinking* the trust surface — provided the bridge period is run with both surfaces hardened simultaneously.

  • During the dual-write bridge period, can you relax ZooKeeper hardening since KRaft is now in charge?
    No. Metadata is dual-written and ZooKeeper is still authoritative for ZK-mode brokers and for rollback. ZK keeps holding live SCRAM/ACLs, so its SASL/TLS/ACL/isolation must stay intact until ZooKeeper is fully decommissioned after finalize.
  • What authorization-consistency issues can derail a ZK→KRaft migration?
    ACLs must translate from the ZK authorizer to StandardAuthorizer with matching principal forms (SSL DN vs SASL name), and super.users must include the controller/broker principals on both sides. Mismatches can deny brokers or controllers mid-migration, stalling or breaking the cluster.
  • Why is migrating off ZooKeeper considered a net security improvement despite a riskier transition?
    The end state removes an entire separately-secured system (ZooKeeper ensemble, its port, JAAS, TLS, ACL migration tooling). Metadata access is then gated solely by Kafka's authN/authZ on the controller listener — a smaller, unified trust surface.

saying these in an interview costs you the question

  • Thinking ZooKeeper can be relaxed or shut down once KRaft controllers start migrating — it stays authoritative until finalize.
  • Ignoring controller-listener security during migration because 'we'll secure it later.'
  • Overlooking ACL/principal translation between the ZK authorizer and StandardAuthorizer.
  • Assuming migration reduces attack surface during the bridge — it temporarily increases it.

context

resolving links…