skip to content

When can you roll back a KIP-866 migration, and what makes the rollback window close?

level: seniorimportance: should knowfreq 40%

answer

  1. rollback OK throughout dual-write
  2. ZK mirror = the safety net
  3. finalize = remove migration flag, stop ZK writes
  4. after finalize: one-way, no return
  5. soak on KRaft brokers before finalize

basics

~20 s

You can roll back any time during the dual-write phase, because ZooKeeper is still a current mirror. The window closes once you finalize the migration by taking the controllers out of migration mode; after that ZooKeeper is no longer written and you cannot return.

solid answer

~40 s

Rollback is possible throughout the dual-write phase — even after some or all brokers are running in KRaft mode — because the KRaft controller has been mirroring every change into ZooKeeper, so ZK reflects current cluster state. To roll back, you reverse the broker roll: restart each KRaft-mode broker back into ZK mode (restore zookeeper.connect, drop process.roles), bring back a ZK-based controller, and decommission the KRaft controllers. Once you **finalize** the migration — i.e., remove zookeeper.metadata.migration.enable from the controllers and restart them in pure KRaft mode — the controller stops dual-writing to ZooKeeper. From that moment ZK begins to drift and is no longer authoritative, so the rollback window is closed; the migration is one-way. Hence the operational rule: do not finalize until you have soaked the cluster in KRaft-broker mode and are confident.

go deeper

for a junior

Know rollback is possible during dual-write and not after finalize.

for a middle

Explain that ZK mirroring is what enables rollback and that finalize stops it.

for a senior

Walk through reversing the broker roll and the soak-before-finalize practice.

for a principal

Reason about silent rollback erosion from ZK-write failures and define explicit abort/finalize criteria.

## Why a rollback window exists at all The whole reason the dual-write phase mirrors metadata into ZooKeeper is to keep a **safety net**. As long as ZooKeeper holds an accurate, current copy of cluster metadata, you can abandon the migration and go back to ZK mode without losing data or metadata. The window is open for exactly as long as that mirror is being maintained. ## What you can roll back from You can roll back: - After provisioning the KRaft controllers but before rolling any broker. - After rolling *some* brokers into KRaft mode. - After rolling *all* brokers into KRaft mode — **as long as you have not finalized**. In all these states the controller is still in migration mode and still dual-writing to ZK. ## How rollback works (reverse the steps) 1. **Revert the brokers**: restart each KRaft-mode broker back into ZooKeeper mode — restore `zookeeper.connect`, remove `process.roles`, and re-add the ZK-mode controller settings. Do this as a rolling restart, one at a time. 2. **Restore a ZK controller**: the cluster goes back to electing a ZK-based controller among the brokers. 3. **Decommission the KRaft controllers**: shut down the migration controller quorum. Because ZK was current, the reverted cluster picks up exactly where it left off. ## What closes the window: finalization **Finalizing** means removing `zookeeper.metadata.migration.enable=true` from the controller configs and restarting the controllers as a *pure* KRaft quorum. At that point: - The controller **stops writing to ZooKeeper**. - ZooKeeper's copy immediately begins to go stale (any new topic, ACL, ISR change is no longer reflected there). - The migration is recorded as complete in KRaft. After finalization there is **no supported path back to ZooKeeper** — the ZK data is no longer trustworthy. You can now safely tear down the ZooKeeper ensemble. ## Edge cases - **Partial finalize confusion**: removing the migration flag from only some controllers is a misconfiguration; treat finalize as an all-controllers action. - **ZK write failures during dual-write** silently erode rollback safety — if the controller has been unable to mirror to ZK, ZK is already stale even though you haven't finalized. Monitor ZK-write metrics before relying on rollback. - **Soak before finalize**: best practice is to run fully on KRaft brokers (still dual-writing) for a period, validate metrics and client behavior, then finalize.

  • Can you roll back after all brokers are already in KRaft mode?
    Yes, as long as you have not finalized. The controller is still dual-writing to ZooKeeper, so ZK is current and you can reverse the broker roll back to ZK mode.
  • What single action makes the migration irreversible?
    Finalizing: removing zookeeper.metadata.migration.enable from the controllers and restarting them as a pure KRaft quorum, which stops dual-writes to ZooKeeper so ZK goes stale.

saying these in an interview costs you the question

  • Believing rollback is impossible once any broker is in KRaft mode — it is possible until finalize.
  • Thinking finalization is reversible — it is one-way; ZK becomes stale.
  • Ignoring ZK-write failures during dual-write, which silently close the rollback window.
  • Finalizing immediately after the last broker rolls without a soak period.

context

resolving links…