A naming convention pins every key of one tenant to a single partition: what does that buy, and what does it cost?
answer
- placement rule sees a group, not keys
- buys multi-key work after the split
- tenant sizes are heavy-tailed
- blast radius changes shape, not size
- the group can only move whole
basics
~20 sIt buys multi-key work over that tenant's keys after the split. It costs placement freedom: the group is placed whole, the largest tenant sizes a node, and one node's loss now removes everything for a few tenants rather than a slice of everyone.
solid answer
~40 sA co-location marker is a designated part of the key name that the placement rule uses instead of the whole key, so every key carrying it lands in the same partition. What it buys is that operations over several of that tenant's keys keep working after the split, along with anything else the store runs as a unit over them. What it costs is that the placement rule no longer sees many keys it can spread; it sees one group it must place whole. Tenant sizes are heavy-tailed, so the largest group sizes the node it lands on, and that tenant's traffic cannot be spread either. The blast radius also changes shape: instead of losing a slice of every tenant, you lose everything for the tenants on that node.
go deeper
Understand the basic bargain: putting related keys in one partition on purpose is what keeps an operation over them working once the keyspace is split. The costs of doing it are a later conversation.
Explain what the convention changes about placement — the rule runs over part of the key name, so a whole group resolves to one partition — and name at least the size and hot-group consequences that follow.
Bring the skew and the blast radius. Say that group sizes are heavy-tailed, that the largest group sizes the node, and that the failure mode has changed from a slice of everyone to everything for a few — then say which you chose and why.
Treat a co-location marker as a placement commitment with a migration cost attached. Set the cap, decide who may introduce one, and be explicit about which customers a single node's loss now takes out entirely.
## What the convention actually does Normally the placement rule runs over the whole key name, which is what makes placement even: keys that differ anywhere land in different partitions. A co-location marker changes the input to that rule. A designated part of the key name — a tenant identifier, a conversation identifier — is used instead of the whole name, so every key carrying the same marker resolves to the same partition, on purpose. This is not universal, and an answer that assumes it is describing one product. Some stores honour such a marker as a naming convention and document it; some expose grouping as an explicit argument on the call rather than as part of the key; and where the spreading is done entirely by the caller's library over independent nodes, the idea may not exist at all. The first thing to establish is whether the tier you are on offers it. ## What it buys - An operation naming several of that tenant's keys keeps working after the split, instead of meeting a cross-partition refusal. - Anything else the store executes as one unit over those keys — a group of steps submitted together, or a script the server runs — stays legal, because its keys are all in one partition. - A single-node design survives the split unchanged for that state, which can turn a rewrite into a rename. - Reasoning stays local: one tenant's state is on one node, so the story of what that tenant lost when a node failed is short. ## What it costs 1. **The group becomes indivisible for placement.** The placement rule no longer sees N keys it may spread; it sees one group it must place whole. Every later placement decision inherits that. 2. **Skew.** Tenant sizes in real systems are heavy-tailed: a handful of customers are orders of magnitude larger than the median. The largest group sizes the node it lands on, and if nodes are provisioned uniformly, the largest group sets the memory ceiling for all of them. Even placement of groups is not even placement of bytes. 3. **Hot groups.** Traffic follows the same distribution. The busiest tenant's load now lands on one node and cannot be spread, because spreading it is exactly what the convention forbids. 4. **The blast radius changes shape.** Before the convention, losing a node removed roughly one node's share of every tenant's keys. After it, losing a node removes everything for the tenants placed there and nothing for the rest. Which is better is not obvious, and saying so is the senior answer: for values that can be recomputed from a system of record, uniform partial loss is a brief slow period spread across everyone, while contained total loss concentrates the pain on a few. For sole-copy state — a claim, a quota, a record that a request was already handled — total loss for a few tenants can be a clean, containable incident with a known customer list, or it can be the worst possible outcome for exactly the customers you least want to affect. 5. **The group cannot be split later.** A live assignment move can move the group whole to a roomier node; it cannot cut it in half. Undoing an over-broad convention means renaming keys, which is a migration under load rather than a configuration change. | | Placement by the whole key name | Placement by a co-location marker | |---|---|---| | Multi-key operation over the group | Refused once the keys spread | Keeps working | | Evenness | Even in keys and roughly even in bytes | Even in groups, skewed in bytes | | One node's loss | A slice of every tenant | Everything for the tenants placed there | | Relieving a hot member | Add nodes and let it spread | No remedy short of renaming keys | ## Sizing the unit of co-location 1. **Co-locate the smallest set the operation actually names**, not the largest set that happens to share an owner. If the operation reads one conversation's messages, the conversation is the unit, not the tenant. 2. **Cap the group.** Decide the largest group the design tolerates, measure against it, and treat a group over the cap as a defect rather than a surprise. 3. **Give outsized members their own arrangement.** A tenant that will not fit is split into sub-groups by something inside it, accepting that cross-sub-group operations go back to being composed in the caller. 4. **Review every new marker.** A marker is cheap to add and expensive to remove, so the moment to argue about it is before it ships. ## When not to use it at all If the only reason for co-locating is that it is convenient to fetch a tenant's keys in one call, and the values do not have to agree with each other, the cheaper answer is one call per key composed in the caller. Reach for the convention when the operation genuinely needs the keys in one place — when it must run as one unit, or when it is a server-side unit that would otherwise be rewritten — and accept the placement consequences deliberately rather than inheriting them.
- How do you pick the unit of co-location?From the operation, not from the org chart. Take the smallest set of keys any single operation names together, and co-locate that. Choosing the tenant because it is the obvious noun is what produces groups nobody can place, and the unit is much harder to shrink later than it was to choose.
- What happens to a co-located group when partitions are moved between nodes while the tier keeps serving?It moves whole or not at all. That is the point of the convention and also its cost: an operator rebalancing a lopsided tier can relocate the group to a roomier node but cannot divide it, so a group that exceeds any single node has no operational remedy short of renaming its keys.
Filing every document of one client in one drawer. Fetching that client's whole file becomes one trip instead of ten, which is the entire point. But the biggest client's drawer overflows while others sit half empty, you cannot relieve it by adding drawers without renaming the files, and if that one drawer burns you have lost one client completely rather than a tenth of everybody.
saying these in an interview costs you the question
- Thinks co-location is free because it only changes key names
- Assumes adding nodes will relieve a node carrying one large group
- Believes every store in this class supports a co-location marker
- Says the group can be split later without renaming keys
- Treats the blast radius as unchanged because the same number of keys is lost
- Chooses the tenant as the unit without checking what operations name