What does a cluster gain by stamping writes to a partition (or queue) with a number raised at each leadership change?
answer
- liveness question becomes a comparison
- raised on every leadership change
- carried on writes and owner maps
- refused at the receiver, not announced
basics
~20 sA number raised at every change of leadership makes a superseded leader's writes rejectable: whoever receives a write compares its stamp with the newest it knows and refuses anything older, without first having to establish whether the sender is alive.
solid answer
~50 sThe stamp turns an unanswerable liveness question into a cheap local comparison. Each time leadership of a partition (or queue) changes, a **generation number** is raised and recorded; writes carry it, and so does the ownership view handed to clients. A node that still believes it leads keeps sending writes under the old number, and any peer or storage layer that has seen the newer one refuses them. Nobody has to notify the superseded leader, reach it, or prove it is dead — it usually discovers the change only when something refuses its work. Within one unit of ownership the number only moves forward, so a returning node cannot argue its way back by re-presenting old authority. Platforms that put one node in charge of a unit fence this way; designs where records live on shared storage enforce the same idea at the storage layer instead.
go deeper
Recall the shape: a counter goes up every time leadership of a partition (or queue) changes, and writes carrying an old counter get refused. That is the whole idea in one sentence.
Explain the mechanics: what carries the number, who compares it, why the comparison is local and needs no contact with the sender, and why it moves only forward within a unit of ownership.
Show that you read the resulting errors correctly — refusals from a node that is no longer the owner are the mechanism working — and that you hunt for write paths, such as an out-of-band repair, that carry no stamp and therefore cannot be refused.
The angle is what your estate relies on when the mechanism is not exposed. Decide what acknowledgement must mean before a team may call a write durable, and where an unstamped path into the data is allowed to exist at all.
## The problem the number solves When the node leading a partition (or queue) goes silent, the cluster cannot tell a crash from a network break — both look like silence. Service is restored by putting another copy in charge, but that leaves an uncomfortable possibility open: the old leader may be alive, may still believe it owns the unit, and may still be accepting writes from clients that can reach it. Asking "is it really dead?" has no answer. The **generation number** replaces that question with one that always has an answer: *is this write carrying current authority?* ## What is stamped, and where The number is a plain counter attached to a unit of ownership, raised each time leadership of that unit changes. It then travels: - **on writes** the leader sends to the copies that must accept them, or to the shared storage that holds them; - **in the ownership view** — the **owner map** — a client caches to know which node to send to; - **in the cluster's own record of membership**, which is what the copies compare against. The comparison is local and cheap: a receiver keeps the newest number it has seen for that unit and refuses anything stamped older. No round trip, no negotiation, no need to reach the sender. | Who sends | Stamp carried | What the receiver does | |---|---|---| | The current leader | current | accepts and acknowledges normally | | A superseded leader that never learned it was replaced | older | refuses; the write cannot become durable | | A client holding an out-of-date owner map | older | its request is refused and it is pushed to refresh | | A node returning after a break | older | must re-synchronise before it can serve again | ## Why the check belongs at the receiver It is tempting to imagine the cluster *telling* the old leader it has been replaced. That message is exactly the one that cannot be relied on — if the network could carry it, the node would probably not have been declared unreachable in the first place. Putting the check where the write lands inverts the dependency: the superseded leader is free to be as confused as it likes, because nothing it does can take effect. It usually learns the truth only by being refused, not by being notified. ## The properties that make it work 1. **It is raised on every change of leadership**, not on every write, so it is cheap and its meaning is unambiguous — a higher number means a later holder of authority. 2. **Within one unit of ownership it does not go backwards**, so a returning node cannot regain authority simply by re-asserting what it remembers. 3. **It is carried, not inferred.** The receiver does not reason about time or reachability; it compares two numbers. 4. **It is per unit of ownership**, so leadership churn on one partition (or queue) does not invalidate work on another. ## What it does not do - It is **not** a timestamp and does not involve comparing clocks between machines. - It does **not** suppress duplicate records or make a consumer idempotent — that is a delivery-semantics concern on the reading side, not a membership one. - It does **not** decide who the new leader should be or when to promote one; it only makes the previous holder's work rejectable afterwards. - It does **not** rescue a write the superseded leader already acknowledged to a client before it was refused downstream — that is why what an acknowledgement waits for matters so much. ## Where designs differ - Platforms that put **one node in charge of a unit** with copies following it stamp writes on the replication path, and the copies are the enforcement point. - Designs that **commit by majority acknowledgement** get much of the same effect structurally: a node on the minority side of a break cannot assemble enough acknowledgements, and the raised number then stops it re-joining as an authority. - Designs with **detached storage**, where records live on shared or remote storage rather than a node's disk, enforce authority at the storage layer — the same idea, a different enforcement point. - **Managed offerings** implement one of these and generally do not expose the number; what you can still rely on is the behaviour, that a superseded owner's writes do not become durable. ## How this shows up to you Operationally you rarely set anything here. You see it as errors that look alarming and are in fact the mechanism working: writes refused as coming from a node that is no longer the owner, a client briefly failing and then succeeding after its owner map is corrected, a returned node discarding a divergent tail before it re-joins. The failure to worry about is the opposite one — a path into the data that carries no stamp at all, such as an out-of-band repair or restore, because nothing on that path can be rejected.
- How does the superseded leader find out it is no longer the leader?By being refused, not by being notified. Its writes carry an older stamp and the copies or storage that must accept them turn them down, which is what tells it to stop and re-synchronise. A notification would depend on the very link whose failure created the situation.
- If a write is refused because it carried an old stamp, has data been lost?Not by itself — the write never became durable, which is the intended outcome. It becomes a loss only if the superseded leader had already told a client the write was accepted. That is why acknowledgement should mean accepted under current authority, not merely written locally.
- Why is the number per partition (or queue) rather than one counter for the whole cluster?Authority is held per unit of ownership, so it has to be checkable per unit. A single cluster-wide counter would make every leadership change anywhere invalidate in-flight work everywhere, and would say nothing about who owns the specific unit a write is addressed to.
- Does this mechanism disappear on a hosted platform?The number usually stops being visible, not the behaviour. You cannot inspect or tune it, and the vendor may enforce authority at a different layer, but a superseded owner's writes still must not become durable. Your side of the contract is to treat an unacknowledged write as unknown.
A landlord re-keys a lock after changing tenants. The old tenant is not chased down and told; their key simply stops opening the door. The check lives in the lock, not in the keyholder's belief — which is the only arrangement that works when you cannot reach the keyholder at all.
saying these in an interview costs you the question
- Thinks the superseded leader must be told before its writes stop counting
- Treats the stamp as a wall-clock time rather than a counter
- Says the number prevents duplicate records reaching consumers
- Assumes the number can go backwards when an old node returns
- Believes every platform enforces it on the same path