During a two-pass roll, why is the agreed internal version raised only after the last broker node carries the new binaries?
answer
- two changes, not one
- binaries first, agreement second
- old members must still keep up
- the first pass is the reversible half
- raising it excludes old binaries
basics
~20 sBecause the agreed internal version fixes how members speak to each other. Held at the old level, a new binary keeps talking the way an old one expects, so any node can still be reverted. Raising it early would leave old members unable to follow their peers.
solid answer
~50 sA version change has two separable parts: the **binaries** each member runs, and the **agreed internal version** — the cluster-wide floor that fixes how members talk to one another. In the first pass the binaries move node by node while the agreed version stays at the old level, so every new member deliberately behaves like an old one on the wire between nodes. That is what makes the mixed-version window work at all, and it is why the first pass is the reversible half: a member that misbehaves can go back to the previous binaries and rejoin a cluster still speaking the old agreement. Only when every member is new does the second pass raise the agreed version, which is a cluster-wide statement that no old member will ever rejoin. Raising it first would strand every node not yet upgraded.
go deeper
Remember the order and the reason: the software on each member moves first, and the cluster-wide agreement about how members talk to each other is raised last, once nothing old remains.
Explain that a new binary holds itself to the old agreement on purpose, that this is what lets mixed releases interoperate, and that raising the agreement early strands un-upgraded members.
Demonstrate treating the two passes as separate changes with a soak between them, and be able to say what evidence you want from the soak before taking the one-way step.
Set the rule for the estate: which pass needs which approval, how long a cluster may sit between passes, and whether the platform couples raising the agreement to other one-way steps.
## Two things are being changed, not one An upgrade feels like one action — "go to the new broker release" — but it is two changes with different reversibility. 1. **The binaries on each broker node.** Local to one machine, changed by stopping it, replacing the software and starting it again. 2. **The agreed internal version.** A cluster-wide floor that says which level of the inter-node protocol all members may assume of each other: how they replicate, how they describe membership, what fields they may put in the messages they send one another. The agreed internal version is deliberately *not* the same thing as the running binaries. A new binary is written to be able to behave like the previous release when the agreed version says so. That capability is the whole reason a cluster can be upgraded without an outage. ## The two passes, in order ``` pass 1 binaries move, agreement held at the old level wave 1 [old][old][old][new] agreement: old <- any node can go back wave 2 [old][old][new][new] agreement: old <- any node can go back wave 3 [new][new][new][new] agreement: old <- still reversible pass 2 agreement raised, binaries unchanged [new][new][new][new] agreement: new <- old binaries can no longer rejoin ``` **Pass one** moves the binaries, one member or one upgrade wave at a time, with the agreed internal version untouched. Throughout, the cluster is a mixture of releases that all speak the old agreement, which is precisely what old members need in order to keep up. **Pass two** raises the agreed internal version once no old binary remains. On platforms that expose this as an operator-set value it is a configuration change rolled out in its own pass; on others the cluster derives the floor from its oldest member and the raise happens implicitly when that member goes. Either way it is the step that converts "a cluster of new binaries pretending to be old" into "a new cluster". ## Why the order is not negotiable - **Raise first, move second** — the agreed version would assert a protocol level that members still running the old binaries cannot produce or parse. Those members fail to replicate or fall out of the cluster, which is an outage caused by the upgrade rather than prevented by it. - **Move first, raise second** — every member understands every other member at all times, because they are all holding themselves to the older agreement. ## What the first pass buys you The first pass is the reversible half, in the specific sense that reinstalling the previous binaries on a member leaves it able to rejoin and to read everything on disk. Nothing in the cluster has yet asserted anything an old binary cannot handle. That is why experienced operators treat pass one and pass two as separate changes, often days apart: - one member is upgraded and watched, then a wave, then the rest; - the cluster is left entirely on new binaries but the old agreement for a **soak period**, carrying real traffic, before anyone raises anything; - only when that soak is clean is the second pass scheduled. The cost of the soak is that the mixed-version window is *not* closed while it runs, and a bounded-skew rule still applies. The benefit is that the one-way step is taken with evidence rather than with optimism. ## What the second pass costs Raising the agreed internal version is normally the point of no return for the release, and it usually pulls two other things with it: | step | reversible? | why | |---|---|---| | new binaries on some members | yes | the cluster still speaks the old agreement | | new binaries on all members | yes | nothing new has been asserted | | agreed internal version raised | no, in practice | old binaries can no longer join or follow | | stored format version raised | no | data on disk is now written in a layout old binaries cannot read | | held-off capability enabled | no | the cluster begins producing state that presumes it | Some platforms couple the last three tightly and some keep them as separate operator actions; where they are separate, each is a decision point of its own, and it is worth knowing in advance which of them your release ties together. ## Where platforms differ Not every platform gives the operator a visible agreed internal version. Some expose it as an explicit cluster-wide value the operator raises; some infer the floor from the oldest member present and raise it automatically when that member leaves; on a rented cluster the provider owns both passes and the tenant never sees the boundary between them. The two-pass *shape* is the constant — move the implementation, then raise the agreement — because it follows from the requirement that mixed releases interoperate, not from any one vendor's design.
- What would actually happen if the agreed internal version were raised first?Members still on the old binaries would be asked to speak a protocol level they do not implement. Depending on the platform they fail to replicate, refuse to join, or drop out of the cluster, so an upgrade meant to avoid downtime causes it. The failure is usually immediate and cluster-wide, not a slow degradation.
- Why do operators leave the cluster on new binaries with the old agreement for days?Because that state carries real production traffic while still being reversible: reinstalling the previous binaries on any member leaves it able to rejoin and read everything on disk. A soak buys evidence before the one-way step. The cost is that the bounded-skew clock is still running, so the soak has to be a decided length, not an indefinite pause.
- Is raising the agreed internal version literally irreversible?On most platforms it is treated as irreversible in practice rather than physically impossible: lowering it again is untested, and any state already written under the new agreement may not be interpretable by the old binaries. Operators plan as if it cannot be undone, because the recovery path is a restore, not a reinstall.
saying these in an interview costs you the question
- Thinks upgrading is one change — replace binaries and you are done
- Would raise the agreed internal version before moving the binaries
- Believes a new binary cannot behave like the previous release
- Calls the whole roll reversible, including after the agreement is raised
- Skips the soak and takes both passes back to back by default