Several hundred consumers and a handful of producers must move to a new schema over days: which side upgrades first?
answer
- who has to tolerate whom?
- only cross-version pairings can break
- the tolerant side moves first
- rollback reverses the argument
- both directions means no order at all
basics
~20 sThe direction that holds decides the order. A backward-compatible change lets the readers go first, since new readers cope with data still written under the old schema; a forward-compatible change lets the writers go first. Full compatibility means any order works.
solid answer
~50 sThe order is not a preference, it is a consequence. During the window both versions are live, and only the **cross pairings** can break: old writer to new reader, and new writer to old reader. **Backward** compatibility covers the first, so you may upgrade **consumers first** and let them read data still being produced under the old schema. **Forward** compatibility covers the second, so you may upgrade **producers first**. **Full** compatibility covers both and imposes no order at all. Two practical riders: the window lasts as long as the slowest straggler, not as long as the deploy; and a **rollback** inverts the argument, because rolled-back readers are old again while some writers have already moved. If neither direction holds, there is no safe order and you are into a coordinated migration, which is a different subject.
go deeper
Remember that during any rolling change both versions run at once, so some records are written by one version and read by another. That pairing is the thing the guarantee is about.
Be able to enumerate the four writer/reader pairings in the window, say which two are free, and map each remaining one onto the direction of compatibility that covers it.
Derive the order out loud from the direction, then volunteer the cases that bite in production: stragglers that keep the window open, replicas mid-deploy, replay of retained data, and the rollback that inverts the requirement.
Own the consequence: decide whether a channel's changes must be rollable, and accept that requiring rollability means requiring both directions and therefore constraining every future schema change on that channel.
## The window, and the four pairings in it A rolling upgrade of a few hundred consumers and a handful of producers is not an event, it is an interval. For some period — hours if you are lucky, days when consumer teams deploy on their own calendars — both schema versions are live at once. Inside that interval every record is written by one version and read by another, so exactly four pairings occur: | Writer | Reader | Needs a guarantee? | |---|---|---| | old | old | No — this was working before the upgrade began | | new | new | No — this is the target state, built together | | old | new | Yes — **backward** compatibility | | new | old | Yes — **forward** compatibility | The two same-version rows are free. The two cross rows are the entire subject, and each direction of compatibility covers exactly one of them. That is why the ordering question has a mechanical answer rather than a stylistic one. ## Deriving the order Upgrade the side that the guarantee protects, and let the unprotected pairing never occur: 1. **Backward compatible only.** New readers can handle old data. Move the **consumers** first; while they run ahead, the producers are still emitting the old version, which is precisely the pairing covered. Once every consumer is across, the producers move and the only live pairing becomes new-to-new. 2. **Forward compatible only.** Old readers can handle new data. Move the **producers** first; the consumers that have not caught up are reading newer data with older code, which is the covered pairing. 3. **Fully compatible.** Both cross pairings are covered, so any order works and no coordination between the two populations is required at all. With a few hundred consumers and a handful of producers, the asymmetry usually pushes teams towards changes that are backward compatible: it is far easier to sequence the small population last than to wait for the large one. ## What makes the window longer than you think - The window closes when the **last** straggler moves, not when the deploy pipeline reports success. One paused consumer keeps the old version alive for as long as it is paused. - Replicas of a single service are themselves a mixed fleet mid-deploy, so even a one-service change has a window. - Anything that replays or reprocesses stored records re-opens the window later, because it presents old bytes to a current reader long after the upgrade finished. - Scaling events can start a process from an older image while the rollout is in flight. ## Rollback is the case people forget A rollback reverses the argument, and this is the failure that catches teams out the first time. Suppose a backward-compatible change and a consumer-first rollout. Halfway through, some producers have already moved, and then the upgraded consumers are rolled back. Those consumers are now **old readers** facing data from **new writers** — the forward direction, which a backward-only change never promised. The consequences are worst exactly when the incident that triggered the rollback is already in progress. The defensible habit is to ask, before the rollout, which direction the *reverse* sequence would require, and to treat a change as safely rollable only if it holds both. That is one of the honest arguments for specifying full compatibility on a channel whose consumers you cannot coordinate. ## When the answer is "neither" If a change satisfies no direction, there is no order that keeps a mixed fleet decoding, and the question as posed has no answer. What follows is not an ordering decision but a migration with a shape of its own — running two contracts at once, or stopping traffic for a cutover — and that is a separate subject with its own trade-offs. What matters here is recognising the situation for what it is, rather than guessing at an order and discovering the gap in production. ## What an interviewer is listening for - That the order is **derived** from the direction, not chosen by preference or convention. - That the candidate names the cross pairings explicitly rather than reasoning about "the upgrade" as a single step. - That rollback, stragglers and replay all come up without prompting, because each of them is a real incident someone has lived through. - That "neither direction holds" is recognised as a distinct answer, not forced into one of the other two.
- Which pairings do you actually have to reason about during the window?Four pairings occur, but two are free: new writer to new reader and old writer to old reader were both already working. Only the cross pairings need a guarantee — old writer to new reader is the backward direction, new writer to old reader is the forward one. Each direction covers exactly one of them, which is why the order follows from the direction.
- The upgraded readers are rolled back mid-rollout. What does that do to your ordering argument?It reverses it. Rolled-back readers are old again while the writers that already moved keep producing the new version, so the pairing is now old reader against new data — the opposite direction from the one the rollout relied on. A change is only safely rollable if both directions hold.
- Does the ordering rule still apply when one process both reads and writes the same type?It dissolves. A process on both sides of the exchange is simultaneously an old reader and a new writer during the window, so no single direction can cover it. Either both directions hold, or you need an ordering guarantee that a mixed fleet cannot give you.
saying these in an interview costs you the question
- Says consumers always upgrade before producers, whatever the change
- Ignores rollback, which reverses the direction relied on
- Assumes the mixed-version window closes when the deploy finishes
- Checks only same-version pairings and never the cross ones
- Treats the ordering as a policy choice rather than a consequence