Rolling Version Upgrades

## What "rolling" actually means A cluster runs the same broker software on several machines. Moving that software to a **new broker release** can be done two ways. - **All at once** — stop every broker node, replace every binary, start everything again. This is what a single-machine deployment does, and on a cluster it is a full outage whose length is the slowest machine's start-up time plus whatever recovery work a node does after being stopped. Writers either block or fail, and their own callers see it. - **Node by node** — take one member out of service, replace its binaries, bring it back, confirm it is healthy and carrying its share of work, then move to the next. At every instant, every member but the one being worked on is still serving. Production clusters use the second shape, and availability is the whole reason. The restart mechanics themselves — how long to wait between members, how you satisfy yourself that a returning node is back in sync, how many may be out at once — are the rolling-restart subject and are assumed here. This subject is the **version change** that the restart carries. ## The price: a mixed-version window From the moment the first node comes back on the new release until the moment the last one does, **two broker releases are live in the same cluster**. That interval is the *mixed-version window*, and nearly everything awkward about upgrading a cluster lives inside it: - nodes of different releases must still talk to one another — replicate records, elect leaders, agree membership and configuration through the metadata service — over a protocol both understand; - a client connects to whichever node it reaches, so one connection may be served by a new node and the next by an old one, and the application must not be able to tell; - anything the new release does *differently on the wire or on disk* must be held back, or an old node meets something it cannot handle. That third point is the reason a version change is normally a **two-pass roll**: the binaries move first while the cluster still speaks the old agreement, and the agreed internal version is raised only once every member is new. | | all at once | node by node | |---|---|---| | availability | none for the duration | full, minus one member at a time | | elapsed time | minutes | hours, or days on a large cluster | | releases live | one | two, for the whole roll | | interop burden | none | both releases must work together | | abandoning it | reinstall everything, restart everything | wave by wave, up to the abandon point | ## What must be true before the first node is touched 1. **Capacity headroom** — the cluster must carry its peak load with one member, or one upgrade wave, absent. A roll started at 95% utilisation finishes as an incident. 2. **Durability headroom** — losing a member must not drop a record's copy set below the floor a write is required to reach, or writes stall while the node is out. 3. **A stated interoperability claim** — the release must say it can run alongside the one you are on. Releases generally support a bounded skew, not an unbounded one, and a cluster left far behind may need an intermediate release on the way. 4. **A decision about the second node** — what you will do if the first one comes back unhealthy. "Continue and hope" is not a plan; "stop, reinstall the previous binaries on that one member, investigate" is. ## Where platforms differ The shape is common to the class, but what an operator *feels* during the window is not. - On platforms that **split a stream into parts** with a leader and follower copies per part, a departing member hands its leaderships to copies elsewhere and reclaims them afterwards, so the roll shows up as leadership churn and brief unavailability per part. - On a **shared-queue** broker where consumers compete for messages, there is no per-part leadership to move; the roll shows up as reconnects and the redelivery of whatever was in flight to the departing node. - On a **rented cluster**, the provider performs the roll on its own schedule, and the operator may see nothing but a series of reconnects. The mixed-version window is still there — it is simply someone else's to manage, and the tenant's job is to be sure its clients survive reconnects. ## The short version Rolling node by node converts an outage into a long interval of degraded-but-serving. The interval is not free: it is an interval in which the cluster is not one system but two releases pretending to be one, and the whole discipline of upgrading is about keeping that pretence honest until the last node is done.