In a messaging system where multiple consumer instances share the label 'consumer group' when reading a topic, how does the broker split the work between them so each message is processed by only one instance in that group?

## The core mechanism The core mechanism starts with how the topic itself is structured: a topic isn't one linear stream, it's split into a fixed number of **partitions**, each an independently ordered, append-only log. Every consumer that joins a consumer group is tracked by a **group coordinator** (a broker-side component, or in some systems a client-side protocol) which owns the job of assigning partitions to group members. The assignment rule is simple but strict: **within one group, each partition is owned by exactly one consumer instance at any given time.** So: - if a topic has 6 partitions and a group has 3 consumer instances, each instance typically owns 2 partitions; - if you scale to 6 instances, each owns exactly 1; - if you scale to 8 instances, 2 sit completely idle because there's nothing left to assign them. This is the mechanism that guarantees "each message reaches exactly one consumer in the group" — it's not achieved by the broker picking a random consumer per message, it's achieved by statically routing an entire partition to one owner at a time. ## Why it exists This exists to solve two problems simultaneously: **horizontal scalability of processing**, and **preserving per-key ordering**. - If the broker just round-robinned individual messages across all group members, you'd get load balancing but lose ordering guarantees — two related events (e.g., two updates to the same order) could be processed out of order by two different consumers running concurrently. - By keying messages into partitions (typically hashed by some business key like customer ID or order ID) and pinning a whole partition to one consumer, you get "ordered per key, parallel across keys" — a sweet spot that plain load-balanced queues don't offer. Consumer groups also give you an important **second axis**: because group membership is what determines message routing, you can have multiple independent groups reading the same topic, and each group gets a complete, independent copy of the stream. This is how publish/subscribe fan-out (e.g., `inventory-service` and `analytics-service` each getting every event) coexists with competing-consumers-style load sharing (e.g., three instances of `inventory-service`, all in the same group, splitting the work). ## The trade-offs The trade-offs center on the partition count being both your **parallelism ceiling** and a semi-fixed operational decision. - You cannot usefully add more active consumer instances to a group than there are partitions — extra instances are pure waste, sitting idle as insurance against failure at best. - **Repartitioning** a topic after the fact (to raise that ceiling) is disruptive: it typically changes the hash-to-partition mapping, which can reorder or reshuffle which consumer sees which key's history, so most teams over-provision partition count upfront rather than resize later. - There's also a real cost to **group membership churn**: every time a consumer joins, leaves, or is considered dead, the group undergoes a **rebalance**, where partition ownership is reshuffled — and in many implementations this is a stop-the-world event for the whole group, pausing all consumption while reassignment completes. ## Failure modes 1. **The rebalance storm.** The headline failure mode is exactly that rebalance storm: if consumers are slow to process a batch and exceed a configured session/heartbeat timeout, the coordinator assumes they're dead, kicks them out, and triggers a rebalance — even though the consumer was just busy, not actually down. This can cascade: the rebalance pauses everyone, work piles up further, the newly-reassigned consumer picks up a partition it's never seen, processes slowly again, times out again, and you get a rebalance loop that tanks throughput system-wide. 2. **The hot partition.** A second common failure is uneven partition-to-key hashing (a "hot partition") where one business key dominates traffic, so one consumer instance is overloaded while its peers in the same group are nearly idle — adding more instances doesn't help because the bottleneck is that one partition, not aggregate capacity. 3. **Offset commit before completion.** A third is a consumer that commits its offset before actually finishing the work (auto-commit misconfigured), so a crash right after commit but before completion silently drops messages rather than redelivering them. ## A worked example A concrete example: a payments platform runs a topic with 12 partitions keyed by account ID, consumed by a "ledger-updater" consumer group running 4 instances — each instance owns 3 partitions, guaranteeing all updates to any one account are processed strictly in order by the single instance that owns that account's partition. A separate "fraud-detection" consumer group reads the same topic independently, seeing every event too, but partitioned differently across its own instance count. When the platform sees ledger-updater falling behind during a Black Friday spike, they scale from 4 to 12 instances (matching the partition count) to use full available parallelism — but scaling to 20 instances would add zero throughput, since 8 of them would have no partitions to own.