skip to content

A colleague proposes putting every key belonging to a customer inside a hash tag such as {tenant:42} so that multi-key commands and Lua scripts keep working after a move to Redis Cluster. What does that buy, and what are the operational risks?

level: seniorimportance: should knowfreq 42%

answer

  1. tag = co-location group = one indivisible slot
  2. tenant sizes are power-law, slots are not splittable
  3. memory skew + single-thread CPU skew
  4. tag the narrowest atomic unit, e.g. {tenant}:order
  5. whale tenants -> dedicated shard/cluster, not a fat tag

basics

~20 s

It buys legal multi-key commands and single-node atomicity per tenant. It costs balance: all of a tenant's data and traffic sit in one indivisible slot, so a large or busy tenant creates a hotspot that resharding cannot fix, since slots can be moved but never split.

solid answer

~50 s

**What it buys:** every key under one tag shares a slot, so `MGET`, `SINTERSTORE`, `MULTI` and scripts over a tenant's keys become legal, and you get real single-node atomicity for that tenant. **What it costs:** the co-location group is unbounded and uneven. Tenant sizes in any real multi-tenant system follow a power law, so one tag can hold gigabytes while thousands of others hold kilobytes. A slot is the unit of migration and cannot be split, so once a tenant outgrows a node, rebalancing has no move that helps - the only remedy is renaming keys and rewriting data. The same slot is one shard's event loop, so a busy tenant also monopolises CPU while other shards idle, and one tenant's big command becomes everyone-on-that-shard's latency. **Better:** tag by the narrowest identity an atomic operation truly spans (`{tenant:42}:order:77`), restructure hot paths into single-key hashes, and split cross-key reads client-side where atomicity is not needed.

code

text · 8 lines
text
> CLUSTER KEYSLOT "{tenant:42}:orders"
(integer) 12055
> CLUSTER COUNTKEYSINSLOT 12055        # run on the owning master
(integer) 8412330                       # one slot, millions of keys -> red flag

# compare masters: skew shows up as lopsided key counts / memory
$ redis-cli --cluster call 127.0.0.1:7000 DBSIZE
$ redis-cli -h node-a INFO memory | grep used_memory_human

go deeper

for a junior

Know that a hash tag forces keys onto one node, and that grouping too much creates an unbalanced cluster.

for a middle

Explain slot indivisibility and both dimensions of skew - memory and single-threaded CPU - as the concrete costs.

for a senior

Argue the granularity rule with evidence: enumerate the paths needing atomicity, tag narrowly, monitor slot key counts, and name the remediation cost of getting it wrong.

for a principal

Frame co-location as a partitioning contract with long-term scaling consequences, including whale-tenant isolation on dedicated shards and the migration project implied by re-keying.

## The proposal in one sentence "Make every key start with `{tenant:<id>}` so all our existing multi-key code keeps compiling." It is a seductive migration shortcut because it turns a wall of `CROSSSLOT` errors into zero code changes. It is also one of the most common ways to build a Redis Cluster that cannot be scaled. ## What you genuinely gain - **Multi-key commands become legal.** `MGET`, `MSET`, `SINTERSTORE`, `RENAME`, `PFMERGE` and friends work across a tenant's keys again. - **Real atomicity per tenant.** Lua scripts and `MULTI/EXEC` can maintain invariants across a tenant's whole dataset, executed on one node with no coordination. - **Locality benefits.** One round trip instead of a fan-out; one node's data structures are warm for that tenant. If tenants were uniform and small, this would simply be good design. The problem is that they are neither. ## Risk 1: the slot is indivisible Slots are the unit of ownership and of migration. An operator can move slot 8461 from node A to node B; nobody can split it into 8461a and 8461b. So the maximum size of a co-location group is the maximum data you are willing to keep on one node - forever. When a tenant's tag exceeds that, your options are all bad: buy a bigger node (vertical scaling, exactly what the cluster was supposed to avoid), move that slot alone to a dedicated node (which fixes memory but leaves that node with one noisy tenant), or rename every key of that tenant and migrate the data live (an application-level project, not an ops task). Note that even shrinking the group later requires rewriting key names, and key names are usually referenced from many code paths and from persisted references. ## Risk 2: skew in size and in traffic Tenant distributions are power laws: the largest tenant is routinely 100x the median. Memory skew shows up as one master hitting `maxmemory` and evicting (or refusing writes) while its peers are half empty; because eviction is per node, the big tenant effectively suffers a smaller cache than everyone else. Traffic skew is worse, because Redis executes commands for a slot on a single thread: a tenant generating 60% of the QPS pins one core while the rest of the cluster idles, and no amount of adding nodes helps. ## Risk 3: blast radius and latency coupling Everything in one slot lives on one master. That tenant's expensive commands - a `SINTERSTORE` over big sets, a script iterating thousands of members, a `DEL` of a huge structure - occupy that node's execution thread, so every other tenant sharing that master sees latency spikes. A single fat tag also concentrates failure: losing that master takes the whole tenant offline at once, whereas a spread-out tenant would degrade partially. ## Risk 4: it hides the real modelling question The tag makes the errors disappear without anyone deciding *which operations actually need atomicity*. Usually a small minority do. By tagging everything, you pay the co-location price on 100% of the keyspace to serve the 5% of access paths that need it, and you lose the ability to tell them apart later. ## Risk 5: it is uncontrolled Braces are just characters; nothing enforces or documents intent. Once `{tenant:42}` is the house style, every new key family inherits it - including growth-unbounded ones like event logs or audit trails that nobody ever intended to co-locate. ## The better shape 1. **Enumerate the access paths that require atomicity or a single round trip.** Typically "the counters and the ledger for one order", not "everything for one customer". 2. **Tag at that granularity.** `{tenant:42}:order:77` co-locates one order's keys; there are millions of orders, so slots stay balanced while scripts still work. 3. **Collapse fields into single-key structures.** A hash read with `HMGET`, a sorted set, a stream - one key is always one slot, and no tag is needed. 4. **Split reads client-side where atomicity is not needed.** Per-node pipelining costs a few parallel round trips and preserves balance. 5. **Accept application-level protocols for anything cross-shard.** Idempotent retries and reconciliation instead of imaginary distributed transactions. 6. **If a few tenants really are enormous**, isolate them deliberately: a dedicated cluster or shard per whale tenant is a legitimate architecture, and much healthier than one accidental hot slot. ## Detecting the problem early Compare per-master memory and key counts, and watch `CLUSTER COUNTKEYSINSLOT` for your largest tags; use `--bigkeys`/`--memkeys` sampling and per-node command rates. Skew is trivially cheap to monitor and extremely expensive to discover after the fact, because the remedy is a data rewrite.

  • The biggest tenant already outgrew its node. What are the realistic remedies?
    Move that slot to a dedicated, larger master to buy time; or re-key the tenant to a narrower tag such as {tenant:42}:order:<id> and migrate the data with dual-write plus backfill; or move that tenant to its own cluster. Resharding alone cannot help, because a slot is the migration unit and cannot be split.
  • How would you decide the right hash-tag granularity up front?
    List the operations that genuinely need atomicity or a single round trip, and take the narrowest identity those operations span - usually an order, a session, or a user rather than a tenant. Then check that the resulting groups are numerous and roughly uniform in size, since numerous small groups spread evenly across slots by CRC16.

Tagging by tenant is like assigning every parcel for one customer to a single loading bay: fine while customers are small, but one warehouse-sized customer jams that bay permanently, and you cannot split a bay in half.

saying these in an interview costs you the question

  • Assuming resharding will fix an oversized slot
  • Treating co-location as free because 'it is all one Redis anyway'
  • Ignoring that a hot slot is one node's single execution thread
  • Tagging by tenant or environment as a blanket migration shortcut
  • Believing key renames are easy to do later once the data is large

context