When moving a broker cluster and its applications to a newer release, which side goes first and why is that not a preference?

## Why the order is forced A client and a broker node settle a wire version at the connect step, and that agreement can contain only what **both** sides can express. The server is therefore the ceiling for every connection into the cluster. Two consequences follow immediately, and together they fix the order: - **Upgrading the server does not break old clients**, because the newer release keeps serving the older request shapes for a stated span. The agreement those clients reach is unchanged, so their behaviour is unchanged. - **Upgrading the client does not unlock anything** until the server can meet it. The new library negotiates down to what the cluster offers, so the work lands with no visible result - and if the library *requires* something the cluster cannot offer, it is refused at connect instead. So: **servers first, clients after.** Not a convention; a consequence of who bounds the agreement. ## What each order actually produces | Order | Immediate effect | Risk | |---|---|---| | Cluster first, then clients | Old clients unchanged; new client capability becomes reachable as each application upgrades | An application already past the cluster's supported span may stop connecting | | Clients first, then cluster | Clients negotiate down; no new capability appears | A client demanding an unavailable revision is refused at connect; teams conclude the upgrade "did nothing" | | Both at once | Nothing gained over cluster-first | Two changes in flight, and an incident cannot be attributed to either | The third row is the one candidates offer most readily and it is the weakest: simultaneity adds no benefit here, because the cluster-first order already leaves every client working. ## The constraint that actually binds In an interview, the order is the easy half. The real material is what stops you executing it: **an unowned client library**. A dependency was pinned years ago, the team that owned the application has been reorganised away, the service still carries real traffic, and nobody has authority to change its dependencies. That library, not your plan, decides how long the cluster must keep serving an old agreement. That turns the question from a per-release one into a **standing** one. Each new server release ships with a support span; each unowned client eats into it; and the day a release finally drops support for that client's shape is the day the application stops connecting - unless someone was made to act months earlier. Practical handling: 1. **Inventory before you plan.** For each application: which client library version, when its connections last re-established, and whether anyone owns it. 2. **Compare that inventory against the span** the target server release states it serves. Anything outside it is not an upgrade task, it is an outage with a date. 3. **Do not let an unowned library veto the cluster upgrade silently.** Name it, cost it, and take the decision explicitly - fund the upgrade, re-home the application, or accept the cut-off. 4. **Re-establish connections deliberately after a client upgrade.** Until a connection is re-made, the old agreement stands and the upgrade is invisible. ## Where platforms differ The rule that the older peer bounds the agreement is general, but its edges are not: - Platforms **state different spans** for how far back a server release serves clients - some several releases, some a time window, some little at all. - Support for the **reverse direction** - a newer client against an older cluster - is much less consistently promised, which is another reason not to lead with clients. - Where the surface is a **versioned remote endpoint** rather than a negotiated agreement, the caller effectively pins its own version, and the ordering question becomes when the old endpoint is retired. - On a **rented cluster**, the provider may upgrade the servers on its own schedule and simply publish a supported client range, in which case your only lever is the client side and the order is decided for you. ## The answer an interviewer is listening for Three beats. First, the rule: the agreement is capped by the server, therefore servers first. Second, the symmetry check: upgrading clients first is not dangerous so much as pointless, with a refusal at connect as the downside case. Third, and the one that separates senior from middle: the order is easy and the **sequencing constraint is organisational** - a client library nobody owns, which is why how-far-behind-may-a-client-be has to be a standing rule rather than a decision taken per release.