Explain the classic ZooKeeper-era two-phase Kafka upgrade using inter.broker.protocol.version and log.message.format.version. Why must the binary upgrade and the protocol bump be separate steps?

**The two version knobs (ZooKeeper-era brokers).** - **`inter.broker.protocol.version`** (IBP): the version of the protocol brokers use to communicate *with each other* (replication, controller/leader-election RPCs, etc.). A broker running new binaries can be told to keep speaking an *older* inter-broker protocol so it stays compatible with brokers that have not been upgraded yet. - **`log.message.format.version`**: the version of the record/message format written to disk and sent on the wire to clients. Bumping it can force a recompression/reformat of records and, critically, can break *old consumers* that don't understand the new format (and can disable the zero-copy optimization when down-converting for old clients). **Why a cluster goes mixed-version.** During a rolling upgrade you replace binaries one broker at a time, so for a window the cluster contains both old and new broker versions. They must keep replicating and electing leaders together. If a new broker immediately started using a newer inter-broker protocol, an old broker receiving those RPCs could fail to parse them — corrupting replication or controller traffic. **Phase 1 — upgrade binaries, hold the protocol.** Before touching binaries, explicitly set in `server.properties` on every broker: ``` inter.broker.protocol.version=<CURRENT_OLD_VERSION> log.message.format.version=<CURRENT_OLD_VERSION> ``` Then rolling-restart each broker onto the new binaries. Now all brokers run new code but negotiate the *old* protocol and write the *old* on-disk format. The cluster behaves exactly as before, and — crucially — you can still **downgrade** by rolling back binaries, because nothing newer was ever written or spoken. **Phase 2 — raise the protocol, then the format.** Once every broker is on the new binaries and healthy, do a second rolling restart that sets `inter.broker.protocol.version` to the new version. After that is stable across the cluster, you typically raise `log.message.format.version` last (often much later, or you skip pinning it on modern versions). Raising the message format is the riskiest/most irreversible step because it affects clients and on-disk data, so it is deliberately done last and separately. **Why separate steps (the core idea).** Two-phase decouples *deploying new code* from *enabling new behavior*. In any mixed-version moment the cluster has exactly one agreed inter-broker protocol — the lowest common denominator — so old and new brokers always understand each other. Phase 1 is fully reversible; only Phase 2 commits you. Doing it in one shot would mean a new broker could speak a protocol the old brokers can't, breaking replication during the window. **Edge cases and gotchas.** (1) **Always read the version-specific upgrade notes** — some jumps require an intermediate IBP. (2) Set the configs *before* the binary roll, not after, or a restarted new broker defaults to the newest IBP it supports. (3) On very modern Kafka, `log.message.format.version` is deprecated/ignored (message format v2 is universal), so the format step largely disappears, but the IBP discipline remains. (4) This whole IBP/message-format dance is the **KRaft predecessor**; under KRaft the equivalent is `metadata.version` managed by feature flags (separate topic).