One in-memory store spans two data halls and the business wants writes accepted in both during a network partition, so what do you tell them and what do you require first?
answer
- restate the ask as a consistency claim
- one keyspace cannot be writable twice
- a third failure domain, or subordination
- price a discarded acknowledged write
- refuse to depend on varying behaviour
basics
~20 sFor one keyspace the request cannot be granted safely: during a split either one hall stops writing, or both write and one side's acknowledged writes are later discarded unmerged. Pick which, deliberately, after inventorying which keys are the only copy.
solid answer
~40 sRestate the ask honestly first. One keyspace, two halls, both writable during a split means two primaries, and on this tier that ends with one side's writes discarded wholesale rather than merged. The real choice is between four postures: a deciding member in a **third failure domain** so one hall can survive alone; a majority pinned to one hall, which is then the only hall that survives alone; manual promotion only, which trades an outage for the guarantee that no machine creates a split brain; or two independent tiers with separate keyspaces, which is a different design and only works if nothing must be global. Before any state depends on the answer, require an inventory of which keys are sole-copy and a number for what a silently discarded acknowledged write costs.
go deeper
Recall that a store spanning two places still has one primary for a given keyspace, and that the side without it cannot accept writes while the link is down.
Explain why both halls writing at once produces two keyspaces that cannot be merged here, and what a tie-breaking member in a third location is for.
Argue a posture with its costs: which hall is subordinate, how long the outage is, and how big the discarded band of acknowledged writes can get before anyone notices.
Own the standing decision. Which keyspaces may live here at all, what a discarded acknowledged write is worth in money, which product behaviours the organisation refuses to depend on, and what on-call is instructed to do instead of lowering the threshold.
## Restating the ask The request sounds like a capacity or availability preference, but it is a consistency claim: *one keyspace, writable in two places that may stop being able to talk to each other.* The lead's first job is to say what that implies rather than which setting to change. During a network partition, two halls both accepting writes for the same keys means the keyspaces diverge. On this tier the divergence is not reconciled when the link returns — the store keeps the current value of each key rather than a version history, and where it treats values as opaque bytes it has no rule for combining two of them, so one side is discarded wholesale. The callers on that side were told their writes succeeded. Granting the request as stated therefore does not buy availability; it buys **silent, unrecoverable loss whose size is the length of the partition.** ## The four postures worth putting on the table | Posture | During a network partition | What it costs | |---|---|---| | Odd membership with a deciding member in a third failure domain | Whichever hall reaches a majority keeps writes; the other is read-only | You must have a third place, and the tie-breaking member has to be genuinely independent of both halls | | Majority pinned to one hall | The primary hall survives alone; the other hall cannot promote | The second hall is not a peer; losing the primary hall is a full outage | | Manual promotion only | Nothing promotes until a human confirms the other side is stopped | A human-length outage, and the human is now the party who can create a split brain | | Two independent tiers, separate keyspaces | Both halls keep serving their own state; they never agree and never need to | Only viable where no state has to be global; it is a different design, not a configuration of this one | The fourth row is worth naming precisely so it is not confused with the request. Two tiers that never share a keyspace have no split brain to have, because there is nothing to diverge. That is frequently the right answer, and it is a different subject from keeping one keyspace writable in two halls. ## What to require before any state depends on this 1. **An inventory of the keyspace by recoverability.** For every prefix, is it derived from a system of record that can rebuild it, or is it the only copy of something — a claim, a quota, a deduplication record, a session? The same discard costs a slow refill in the first case and a double-charged customer in the second. 2. **A price on a discarded acknowledged write.** Not a feeling, a number: what one silently lost claim or deduplication record costs, times a plausible partition length at the write rate. That number is what justifies either the third failure domain or a flat refusal to keep that state here. 3. **A named minority-side behaviour, verified on the product you actually run.** Whether the isolated primary refuses writes below a healthy-copy threshold, and how quickly, is not a property of the class. Some products offer such a gate, some offer none and serve the isolated side indefinitely, and some managed offerings resolve it from a control plane outside the data plane that can cut the node off without its cooperation. 4. **A measured rediscovery window.** How long after a promotion the last caller is still addressing the old node. It is regularly the longest of the three clocks and the least instrumented. 5. **A standing rule for the incident.** What the on-call engineer does when nothing promotes because no side has a majority. Without a rule, the reaction is to lower the threshold or promote by hand on the smaller side, and that is how most real split brains are manufactured. ## The guarantee to refuse to depend on A useful organisational habit on this tier is to name the guarantees you will not build on because they vary across the products you might buy. The minority side's self-policing is the clearest candidate: if the design only works when the isolated primary stops itself promptly, it is coupled to a behaviour that differs between stores, differs between versions, and can be turned off by a configuration change nobody reviews. A design that is merely *degraded* when that behaviour is absent is portable; one that is *incorrect* without it is not. ## The answer that lands Do not answer with a topology. Answer with the trade, then the question that decides it: *which is worse for these particular keys, a refused write or an accepted write that will be thrown away?* Then show that you know the honest options are to buy a third failure domain, to declare one hall subordinate, or to stop keeping that state in a volatile tier at all — and that none of them is the thing that was asked for.
- Why is a tie-breaking member in a third failure domain preferable to simply adding another node in each hall?Because adding one node per hall keeps the membership even and keeps both halves equal, so an even split still promotes nobody. The tie-breaker works by being somewhere that fails independently of both halls, which is what lets exactly one hall form a majority during a split. It holds no keyspace and serves no traffic, so it is cheap, but it must genuinely not share the failure mode it is breaking.
- The team proposes manual promotion only, so no machine can ever create a split brain. What do you check?That the human has a way to confirm the old primary is stopped rather than merely unreachable, and that the runbook says what to do when they cannot. Manual promotion removes the automatic split brain and relocates the decision to the person least able to see the whole network at three in the morning. It also lengthens the outage to human scale, which has to be acceptable for the state involved.
saying these in an interview costs you the question
- Answers with a topology diagram instead of naming the trade being made
- Promises both halls stay writable and the keyspaces reconcile afterwards
- Treats the minority side stopping itself as a property of every product
- Adds one node per hall and calls the even membership highly available
- Confuses two independent tiers with one keyspace writable in two places