skip to content

A sharded index's copies are being replaced one by one; what must hold for old and new members to keep replicating to each other?

level: seniorimportance: must knowfreq 50%

answer

  1. the two versions are each other's clients
  2. both directions, not just the new one
  3. no one-way on-disk upgrade at start-up
  4. negotiate down, never refuse an older peer
  5. write the new format as a second step

basics

~20 s

Both directions must work at once: the new member reads what the old ones wrote, and the old ones tolerate what the new one writes. They are each other's replication source for the whole rollout.

solid answer

~50 s

In this window the two versions are not merely serving the same clients; they are each other's clients, feeding one replication stream and reading stores of one shared format. So the new version must open what its predecessor left and accept what its still-old peers send, and it must not emit anything the old members cannot parse — the rollout can halt at any copy, and can be reversed. That rules out a new version whose first act on start-up is an irreversible on-disk upgrade, and it rules out a peer protocol that refuses older versions instead of negotiating down to what everyone present speaks. The safe shape is to ship the new capability dormant: every member reads the new thing, nobody writes it, and writing is switched on as a deliberate second step once the whole group is new.

go deeper

for a junior

Hold on to the shape: for part of the rollout, some copies run the new version and some the old, and they are talking to each other the whole time.

for a middle

State both obligations explicitly and give the mechanism for each — ignorable additions in the data, negotiated-down versions in the protocol.

for a senior

Apply the test that matters: if the rollout stops after one copy and stays stopped for a week, does the group still work and can that copy go back?

for a principal

Decide where the burden sits. A group that verifies capabilities before enabling a new format costs a release of delay and removes a whole class of outage; a group that trusts the operator does not.

During an ordered one-at-a-time replacement, a replicating group spends the whole rollout — often hours — running two versions of itself simultaneously. This window is the reason the rollout is interesting, and it is materially different from a mixed-version window in a set of interchangeable, stateless copies. ## Why this window is not the usual mixed-version window When interchangeable copies are replaced in batches, the two versions share **clients**. They answer the same requests, and the compatibility question is about what a caller sees. Here the two versions share far more: - **They are each other's clients.** Every member sends and receives a replication stream from members that may be on the other version, in both directions, continuously. - **They share durable state that outlives both.** Whatever gets written into a store during the window is still there after the rollout ends, and after any decision to go back. - **They cannot be separated by routing.** You cannot keep old traffic on old members; the group has to agree with itself regardless of who is talking to it. ## The two obligations, and why both are mandatory 1. **The new version must handle what the old members wrote and are still writing.** It opens a store its predecessor left, and it receives a live stream from peers running the previous code. 2. **The old members must tolerate what the new version writes.** This is the one that gets skipped, and it is not optional: those old members are consuming the new member's output *right now*, and the rollout may stop at any copy or be reversed before it finishes. If only the first obligation is met, the group works in the direction someone tested and breaks in the direction nobody did — usually as a member that silently drops records it cannot parse, which is worse than a crash because it is invisible until you go looking for the data. ## What that means for the data format - **Additions must be ignorable.** A field an older member does not know about has to pass through it rather than fail its parser or be dropped on rewrite. - **Nothing may be renamed or repurposed in a single step.** Read both spellings for one release, write the new one only when the whole group is new. - **No irreversible on-disk upgrade at start-up.** A new version that rewrites its store into a format its predecessor cannot open makes the very first replaced copy a one-way door, so the group can no longer go back even though only one member has moved. - **Switch on writing as a second step.** The reliable shape is two stages: first a version that *reads* the new format everywhere and writes the old one, then, once every member is new, a deliberate flip that starts writing it. ## What that means for the peer protocol - **Negotiate down, do not refuse.** A member that will only speak the newest version of the protocol partitions the group at the first replacement, and the split lasts until the last copy is replaced — during which the group has no majority anywhere. - **Version each message, not the connection.** New message types must be optional, and a member that receives one it does not recognise should ignore it rather than drop the peer. - **Do not change the meaning of an existing message.** Reusing a field for a new purpose is undetectable to an old peer, which will act on the old meaning with full confidence. - **Assume the window is long.** It is the number of members multiplied by per-member catch-up time, and it can be hours; it is not a few seconds of overlap. ## The question to ask before the rollout starts A useful test, asked of the version you are about to roll: *if this rollout stops after one copy and stays stopped for a week, does the group still work, and can that one copy go back?* If the answer to either half is no, the compatibility work is not done, and the ordering of the rollout will not save you — it only controls *which* members are on which version, never whether the two versions can live together. Designs differ in how much of this they do for you. Some systems negotiate a protocol version explicitly at every handshake and refuse to write a new format until every member reports the new capability; others leave the entire burden to the application and will happily let a single replaced member write something its peers cannot read. Knowing which of those you are running is part of knowing whether your rollout is safe.

  • Why is a new version that upgrades its store's format on first start-up so dangerous here?
    Because the very first replaced copy becomes a one-way door. Its store can no longer be opened by the previous version, so that member cannot go back, and if the rollout is halted the group is permanently mixed with one member that cannot be reverted. The upgrade has to be an explicit, separate action taken once every member is new.
  • How long does this window actually last?
    As long as the rollout, which is the member count multiplied by how long one member takes to stop, start and catch up — commonly hours for a group holding real data, and indefinitely if the rollout is deliberately held part-way. Treating it as a brief overlap is what leads people to skip the compatibility work.
  • What does it look like when only one direction of compatibility was tested?
    Usually silent data loss rather than a crash. The new member emits records with something extra in them, the still-old members parse what they recognise and drop the rest when they rewrite, and the group looks healthy the entire time. It surfaces later as records that are missing exactly one field, on exactly the members that had not been replaced yet.

It is like replacing a ship's crew one watch at a time: the new hands may bring better instruments, but until the last old hand is ashore, every log entry still has to be written in the notation the whole crew can read.

saying these in an interview costs you the question

  • Only checks that the new version can read old data
  • Lets the new version rewrite its store's format on first start
  • Assumes the window lasts seconds, not hours
  • Has the new version refuse peers on the previous version
  • Reuses an existing message field for a new meaning in one release
  • Believes the fixed order removes the need for compatibility