Spreading the Tier

Acknowledgment means one node has the write in memory, nothing more. Until it propagates, the write exists in exactly one place, and if that node is lost the write is simply gone.

A replica holds a copy of the keyspace but is not the address that accepts writes, so writes resume only after some deciding party promotes one of them to primary. That promotion is a deliberate step, and it takes time.

Assignment must be a deterministic function of the key, evaluated identically by every caller. If two callers disagree about where a key belongs, each reads and writes its own copy on a different node and neither sees the other's.

A replica answers from its own copy, which trails the primary by the propagation lag, so a read issued inside that window returns the value the key held before the write. The write is not lost, only not yet visible there.

Three reworks are honest: one call per key composed in the caller, folding the four values into one entry under one key, or accepting the operation is no longer one unit and saying what a reader does with a mixed view.

Each lookup turns into a cross-region round trip - tens of milliseconds instead of a fraction of one - which can exceed the cost of the work the tier was avoiding, making the remote path slower than having no tier.

Waiting for a copy narrows the un-propagated window and adds a network hop to every write. It does not close the window: a write only one copy holds is still lost if that copy is not the one promoted.

A failover outage is the sum of three windows: detection, before anything concludes the primary is gone; promotion, while a replica is reconfigured; and rediscovery, until the last caller stops addressing the dead node.

Three shapes exist: the caller holds a copy of the partition map, an intervening proxy holds it, or a directory is consulted per lookup. They differ in hops, in who must learn of a change, and in who can be stale.

A majority gate guarantees that only one side of a network partition can promote, because two separated sides cannot both hold more than half of one membership. Without it, both halves promote, both accept writes, and the two keyspaces diverge.

It depends on whether the node reached checks ownership. Where it does not, a misrouted read returns an ordinary absence and the caller treats live state as gone; where it does, a reply names the right node to follow and learn from.

The tier keeps the current value of each key and no history of how it got there, and where values are opaque bytes there is no merge rule. So the losing node discards its keyspace, copies the winner's, and its acknowledged writes vanish.

Fan-out across copies buys read request capacity and nothing else. Writes still land on one primary and are then propagated to every copy, and each copy holds the whole keyspace, so neither write throughput nor memory headroom improves.

It buys multi-key work over that tenant's keys after the split. It costs placement freedom: the group is placed whole, the largest tenant sizes a node, and one node's loss now removes everything for a few tenants rather than a slice of everyone.

Both values are locally derived views of one shared system of record, so neither is authoritative and there is nothing to merge. Making them agree would put a cross-region exchange back on the write path.

Neither entry is a copy of anything, so two independent tiers hold two different facts about one user: the quota is granted once per region, and a retry landing in the other region finds no record and is treated as new.

A minimum-healthy-copies write gate converts a silent loss risk into a loud refusal: writes are accepted only while enough copies are current. It narrows the un-propagated window and does not close it.

Multiply the tail propagation lag at peak write rate by that write rate to get a count of writes at risk, then say what those writes were. The count alone is not an answer; the state they held is.

Promotion only moves which node accepts writes; it does not reach into callers. A service that resolved the write address once at start-up and never again keeps talking to the dead node, however clean the failover was.

A stall longer than the detection window is indistinguishable from death, so the tier pays a full outage for nothing and must then demote a node that returns believing it leads. Size the window above the measured stall tail.

Three parties can own the decision: the surviving peer nodes, a set of dedicated observer processes, or a controller outside the data plane. A single observer is never enough — it cannot tell a dead node from its own severed link.

Reads whose correctness is the presence or absence of a key at this instant cannot be answered by a lagging copy: route that class to the primary. Every other read gets a named staleness budget, measured against propagation lag rather than assumed.

Because the majority side cannot reach it to tell it anything. Safety depends on the isolated primary acting against itself on a local timer, refusing writes or demoting, since a promotion is very likely already happening on the other side.

Rule: no design may assume one address space. Every call naming more than one key is classified - co-located under a capped group, split, folded, or explicitly accepted as non-indivisible - and reworked before the split.

Decide by who bears the cost of a change, not by hop count. Caller-side maps spread correctness across every client version deployed; a proxy concentrates it into one component you run; a directory is authoritative but priced per lookup.

For one keyspace the request cannot be granted safely: during a split either one hall stops writing, or both write and one side's acknowledged writes are later discarded unmerged. Pick which, deliberately, after inventorying which keys are the only copy.

Three separate losses, not one: derived entries for the arriving population are absent and get rebuilt; sole-copy state held only in the lost tier is unrecoverable; and if that region accepted the writes, the write path is a second, larger failure.

Standardise the number, not the mechanism: each tier declares the un-propagated window it accepts and proves it with a real promotion. Refuse to depend on guarantees the fleet's products implement differently.

Fan-out only buys capacity for reads you will answer from a lagging copy, so each pin returns part of it. Measure which classes are pinned and why, then change what the tier holds or how it is split rather than adding copies.