Should a cluster node that has lost contact with the coordination role stop serving writes on its own, or rely on later refusal?

## The decision, stated plainly A node leading a partition (or queue) loses contact with the **coordination role** — the part of the cluster that records membership and decides who leads. It does not know whether the rest of the world is gone or whether it is. Two policies are available, and they are ends of a spectrum rather than a binary: - **Self-demotion**: after a bounded period of silence, the node stops serving writes for the units it leads, on its own initiative. - **Refusal-only**: the node keeps serving, and correctness is preserved downstream — its writes carry an out-of-date generation number and are refused wherever they would have to become durable. Both are sound. Neither lets a superseded leader's records enter the accepted history. The choice is about **how long a client can be told something that later turns out not to have happened**, and about how readily the estate gives up capacity. ## What each one costs | | Self-demotion after silence | Refusal-only | |---|---|---| | Time a client can be misled | bounded by the silence period | until a write is refused or times out | | Behaviour during a brief disturbance | the node steps aside, and service for its units pauses | the node keeps serving and usually rides it out | | Failure mode to fear | a healthy node removing itself during a monitoring wobble | work accepted locally for a while and then discarded | | Ease of reasoning in an incident | high — a silent node is also an idle node | lower — a reachable node may be doing nothing useful | | Dependence on client discipline | lower | high | The asymmetry worth naming: self-demotion converts an uncertainty into a small, deliberate availability loss, while refusal-only converts it into a larger uncertainty that somebody else — the client author — has to handle correctly. ## What you can actually choose Be honest about the level at which this is decided: 1. **The platform's design** usually fixes it. Some products demote on lost contact; some rely on rejection at the replication path; designs with detached storage enforce it at the storage layer and the node's own opinion matters even less. 2. **A hosted offering** chooses and may not document it. What you can still hold it to is the outcome — a superseded owner's writes must not become durable. 3. **What is genuinely yours** is the contract you impose on everything that writes, and the paths you allow to exist at all. So a policy written as "our brokers shall self-demote within N seconds" is often unimplementable. A policy written as what teams must assume is always implementable. ## The policy worth writing - **Acknowledgement is the only evidence.** A write is durable when the cluster acknowledged it under current authority. Locally accepted is not durable, and a log line on the writer is not an acknowledgement. - **An unacknowledged write is unknown, not failed.** Every writing service must be safe when the record turns out to exist after all, and safe when it does not. - **Retry means re-resolve.** A retry that does not re-resolve which node owns the unit is a defect, not a tuning choice. - **No unstamped path into the data.** Repair tools, imports and restores bypass the ownership check by their nature; each needs a named owner, a reason, and ideally a way to run them only when the cluster is quiet. - **State the silence tolerance per stream class, not per estate.** A stream carrying payment instructions and one carrying page views do not deserve the same answer about how long a node may keep serving in the dark. ## Where a principal earns the title The interesting judgment is not picking a side; it is noticing that the question is usually settled by a purchase rather than by a standard, and then defending the part that is still in your hands. Two moves do most of the work. The first is separating the streams whose provisional writes would be expensive from those where they would not, and accepting different behaviour for each rather than one estate-wide number that satisfies nobody. The second is refusing to let an ambiguity be resolved by an operator under pressure — an incident is exactly the moment when somebody proposes to force a change on a node that is merely unreachable, and whether that is ever allowed, and who may authorise it, belongs in a standard written on a calm day. The failure this policy prevents is mundane and common: a team ships a writer that logs success on a local acceptance, an incident isolates a node for ninety seconds, and months later nobody can explain a gap in a stream that every dashboard says was written.