A tier whose nodes each hold a share of the keyspace, not a copy: what limits an operation naming three keys?

## Two different meanings of "we added nodes" Adding machines to a volatile tier means one of two things, and the two behave in opposite ways. In the first, each added node holds **a copy of the whole keyspace**: every node can answer for every key, and what you gain is read capacity and something to promote when the primary is lost. In the second, the keyspace is **cut into partitions**, and each node holds a share of it: a key belongs to exactly one partition, and a partition is served by one node at a time. Only the second arrangement raises this problem. It is worth saying out loud in an interview, because "we scaled the tier out" describes both and they fail in opposite directions — copies serve answers that are behind and lose recent writes on a promotion, while pieces refuse multi-key work and misroute when a caller's copy of the partition map is out of date. ## Why one operation cannot reach two partitions A node executes an operation against memory it owns. For one operation to read or write keys living on two nodes, something would have to fetch the remote values, combine them, and hold both nodes still while it did so. Stores in this class deliberately do not put that in the path. Their whole value is that an operation is a few microseconds of work against one machine's memory with no coordination in it; making every operation able to span nodes would make the common case pay for a case most workloads can design away by choosing key names better. Making one change across independent parties atomic is a genuine subject with genuine protocols — it is simply not what this tier offers. So the rule is blunt: **an operation that names several keys can run only if every one of those keys is in one partition.** Anything else the store executes as a single unit inherits the same rule — a group of steps submitted together, or a script the server runs on your behalf — and the store can only enforce the rule if it is told every key the unit will touch before it begins. ## What the caller actually observes — and this varies | How a key is resolved to a node | A multi-key operation spanning partitions | What the caller sees | |---|---|---| | A map held by the caller | The store declines it | A cross-partition refusal: an error on a path that worked yesterday | | An intervening proxy | Some decline; some read each partition and compose | Either a refusal, or a reply that was never one indivisible operation | | A directory consulted per key | Usually there is no such operation to offer | The caller resolves each key and issues one call per key | | Independent nodes, spread by the caller's library | Not offered at all | Nothing changes: the library always issued one call per key | The row that catches people is the second. A composed reply is neither a refusal nor an indivisible read: the values came from different nodes at slightly different moments, so the group can be observed part-old and part-new. A design that quietly leans on that behaviour also breaks if the routing shape is ever changed. ## The four honest responses - **Co-locate deliberately.** Put a marker in the key names that the placement rule uses instead of the whole key, so related keys land in one partition on purpose. The operation keeps working; the group can then never be spread. - **Split the call.** Issue one call per key and compose in the caller. It works everywhere, and the values are no longer observed at one instant. - **Fold the group into one entry.** One key is one partition by construction. The entry grows, and on stores that only hand back opaque bytes every change becomes a read of the whole value and a write of the whole value. - **Accept that it is no longer one unit.** Sometimes the right answer is to decide explicitly what a reader does when it sees part of the group changed and part not. ## What the constraint is not It is not about load: the call is not refused because the tier is busy, and adding nodes does not relieve it — adding nodes makes it more likely, because related keys spread further apart. It is not about the size of the values. And it does not arise at all on a tier of copies, where every node holds the whole keyspace: there nothing is refused for this reason, and the caller's problem is an answer that is behind, which is a different subject entirely. Naming the arrangement before answering is most of the mark on this question.