An in-memory store holding both re-derivable values and sole-copy records serves all reads from replicas; which reads must return to the primary, and how do you bound the rest?

## The classification is the answer Once reads are fanned out across copies, every read is answered from a snapshot that is some amount of time old. The engineering question is not "is this acceptable?" in the abstract; it is "which reads is it acceptable for, and how do I know?" The useful axis is what the caller does with the answer. | Read class | What an old answer costs | Where it should go | |---|---|---| | Advisory — rendered, displayed, counted for a human | A slightly old screen | A copy, with a named staleness budget | | Re-derivable — the value has another home | Extra work, a slower path | A copy; the system of record settles ties | | Decision on presence or absence, right now | A wrong decision, not an old one | The primary | | The caller's own read-back after its own write | Confusing behaviour, usually a bug report | The primary, for a bounded interval | The third row is the one worth naming carefully, because it is a whole family: a check on whether someone else currently holds a claim on a resource, a decision about whether an allowance has been spent, a lookup asking whether a unit of work has already been recorded as done. In each of them the answer is the *existence* of a key at this moment, and existence is precisely what the lag moves. Those reads do not get a little bit wrong as the lag grows; they get the decision backwards. How each of those records should be designed is a workload question in its own right — what belongs here is only the routing consequence. ## Why the state's other home matters The same lag produces an annoyance or an incident depending on a fact about the keys, not about the tier: whether anything else can reproduce the value. - If the value is derived and another system holds the truth, an old read costs recomputation or a slightly stale render. - If the tier holds the only copy of the record — a claim, an allowance, a note that something already happened — nobody can correct the answer, because there is nothing to correct it against. So the classification has to be applied per key family, not per tier. A single store routinely holds both kinds, which is what makes "just point reads at the copies" such a durable source of incidents. ## Pin the class, not the caller Pinning individual callers feels cheaper and ends badly. A caller is a deployment artefact: it gets split, renamed, replaced by two services, and the reasoning that justified the pin does not travel with it. The alternative is durable: - A read class — "is this claim held?", "is this allowance spent?" — is a property of the decision, and stays true wherever the code lives. - A class is **reviewable**: someone can ask whether it genuinely needs the primary, and the list is short enough to argue about. - A list of pinned callers only ever grows, because nobody who did not write the code dares unpin it. ## Bounding the staleness, honestly For everything else, the discipline is to convert a hope into a measurement: 1. **Measure propagation lag continuously** for the copies you route reads to, as time behind or as un-propagated work. 2. **Name a budget** per read class — the amount of staleness that class tolerates. Derive it from what the answer is for, not from what the tier currently achieves. 3. **Compare the tail, not the median.** The defensible bound is what the distribution does at the high percentiles, because those are the reads that generate the reports. 4. **Decide the behaviour on breach.** When the measured lag exceeds the budget, the read path must do something — route that class to the primary, or fail the read — because an unenforced budget is an intention. Exactly how lag is exposed differs across stores in this class, which is a reason to build the measurement into the read path's own view of the world rather than depending on a particular reporting surface. ## What bounding cannot do A staleness budget changes how *often* a read is behind, not whether it can be. That is fine when the cost is a stale render and useless when the cost is a wrong decision: a key that appeared or vanished within the budget is still missed. Which is why decision reads are routed rather than budgeted — the two remedies are not interchangeable and mixing them up is the most common error here. ## The bill Every read sent back to the primary is a read the copies do not serve, so it hands back part of the capacity the fan-out was bought for. That is not an argument against pinning; it is the reason to classify precisely rather than defensively, and it is the number to put in front of anyone who proposes more copies as a fix. ## Two independent reasons to route a read to the primary — a read class whose answer is a decision, and a bounded pin after this caller's own write. The pin lookup reads a default so an unwritten key takes the copy branch. This shape is expressible only where the caller chooses the route; behind an intervening proxy the same policy has to live in the proxy instead ``` # caller-side routing decision on write(key, value): primary.put(key, value) pinned_until[key] = now() + pin_window on read(key, read_class): if read_class is "decides-on-presence": return primary.get(key) # never a copy if now() < pinned_until.get(key, default=0): return primary.get(key) # this caller's own write return some_replica.get(key) # advisory: budgeted ```