A partitioned topic is keyed by user ID to preserve per-user event ordering, but one 'celebrity' user generates 100x the event volume of a typical user. What problem does this cause, and what are reasonable ways to mitigate it without abandoning per-user ordering entirely?

## Why a hot key concentrates on one partition **Key skew** is the direct consequence of the same mechanism that provides per-entity ordering: since a partition key deterministically maps to one partition via `hash(key) % numPartitions`, every event for a given key is, by design, funneled to the same place. That's exactly what you want for ordering, but it means the partitioning scheme has no way to spread load within a single key's traffic — partition count only helps distribute load across different keys. If one key (a celebrity user's account, a mega-tenant in a multi-tenant SaaS product, a single stock symbol during a trading spike) generates disproportionate volume, all of that volume lands on one partition no matter whether the topic has 6 partitions or 600. ## The symptom, and why it hides The observable symptom is a **hot partition**: its consumer-side lag (the gap between the latest produced offset and the last committed offset) climbs steadily while every other partition in the same topic stays near zero, because the one consumer instance assigned that partition is saturated while its siblings are comfortably idle. This is dangerous precisely because aggregate lag metrics across the whole topic can look acceptable — if 23 of 24 partitions are healthy and only 1 is falling behind, a topic-level average can mask a user-facing problem (that one entity's downstream state is stale or delayed) entirely. Teams that only alert on aggregate lag, not per-partition lag, routinely miss this until a customer complains. ## Three mitigations, and what each gives up The trade-off in every mitigation is the same: you're relaxing the guarantee that all of one entity's events are strictly ordered on a single partition, in exchange for spreading that entity's load across more resources. | Mitigation | What it means for the hot entity | |---|---| | key salting/sub-keying | restores parallelism for the hot entity but breaks strict cross-sub-partition order | | isolating known hot keys onto dedicated infrastructure | keeps strict single-partition ordering for the hot entity | | increasing total partition count, when the skew is moderate rather than extreme | the hot key's ceiling is now whatever one partition/one consumer instance can sustain, as a known, monitored limit | 1. **A common technique is key salting/sub-keying**: append a bounded, deterministic-or-round-robin suffix to the hot key (e.g., userId becomes userId#0 through userId#7), producing 8 sub-streams for that one entity that spread across up to 8 partitions instead of 1. This restores parallelism for the hot entity but breaks strict cross-sub-partition order — the consumer side must now either tolerate approximate ordering (fine for, say, an analytics/metrics pipeline) or actively reassemble true order using an application-level sequence number or timestamp embedded in each event, essentially reimplementing ordering in the consumer instead of relying on the log for it. 2. **A second approach is isolating known hot keys onto dedicated infrastructure**: detect (via monitoring or a pre-known list, like verified/celebrity accounts in a social platform) which keys are disproportionately large and route them to a separate topic sized and partitioned specifically for that traffic pattern, or even give each known hot key its own dedicated partition via a custom partitioner. This keeps strict single-partition ordering for the hot entity while not making every other, normal-volume key suffer from a partitioner tuned around the outlier. 3. **A third, cheaper mitigation when the skew is moderate rather than extreme** is simply increasing total partition count and relying on a well-distributed key space for the rest of the traffic, while accepting that the hot key's ceiling is now "whatever one partition/one consumer instance can sustain" as a known, monitored limit — appropriate when the hot entity's absolute volume, while large relative to peers, is still within what a single fast consumer can handle (e.g., a well-optimized consumer doing simple filtering/routing may sustain far more throughput per partition than one doing heavy per-event computation). ## The failure mode to watch for The failure mode to watch for in all of these: sub-keying without a corresponding consumer-side reordering strategy silently reorders a hot entity's events for anyone downstream who assumed strict order, which can be worse than the lag problem it solved — e.g., an "account suspended" event processed before a "transaction" event from the same sub-key split, because they landed on different sub-partitions and were consumed at different rates, causing a transaction to post to a supposedly-suspended account. ## Where it shows up A concrete real-world scenario: ride-sharing and social-media platforms both commonly hit this with celebrity/power-user accounts or extremely active drivers/regions — a location-update or activity-event stream keyed by user ID can have one influencer account generating orders of magnitude more events than a typical user during a viral moment; production systems typically respond by combining per-partition lag alerting (to catch it early) with a sub-keying or hot-key-isolation strategy specifically reserved for identified outliers, rather than redesigning the partitioning scheme for the entire, otherwise well-behaved key space.