In a Kafka-style consumer group, what is a 'rebalance,' and what effect can it have on message ordering and processing guarantees while it's happening?

## What a rebalance is A **consumer group** is Kafka's mechanism for letting multiple consumer instances split the work of reading a topic's partitions, with the constraint that each partition is owned by exactly one consumer in the group at a time (so within a partition there's still a single reader maintaining order). A **rebalance** is the process by which the **group coordinator** (a broker) reassigns which consumer owns which partitions. Rebalances are triggered by group membership changes: - a new consumer instance joins (e.g., during a rolling deploy or scale-out); - an existing one leaves gracefully; - one is declared dead because it missed its heartbeat/session-timeout window (e.g., a long GC pause or a slow poll loop); - or the topic's partition count changes. ## The two protocols Mechanically, Kafka's newer cooperative/incremental rebalancing protocol (via `CooperativeStickyAssignor`) improves on the older eager rebalance protocol: | Eager | Cooperative, incremental | |---|---| | every member of the group revokes all of its currently assigned partitions before the coordinator computes and hands out a brand-new assignment | only the partitions that are actually moving get revoked and reassigned, while consumers keep reading their unaffected partitions throughout | | this is a stop-the-world event for the entire group — no partition in the topic is being consumed by anyone for the duration of the rebalance, even partitions whose ownership doesn't actually change | the assignor tries to keep partitions with their previous owner where possible ("stickiness") to minimize churn | ## Why the mechanism exists This mechanism exists because you need dynamic membership to make consumer groups elastic and fault-tolerant: instances should be able to scale in and out, and a crashed instance's partitions need to be picked up by a survivor without manual intervention, or the group would silently stop making progress on those partitions. Rebalancing is the coordination protocol that keeps "every partition owned by exactly one live consumer" true as the group's membership changes. ## The trade-off The trade-off is availability/liveness during the rebalance window versus complexity. - **Eager rebalancing** is simple to reason about but costs a full-group pause — with a group of many consumers and a slow rebalance (e.g., large state to restore, or consumers with slow `onPartitionsAssigned` callbacks), this pause can stretch to seconds or tens of seconds, directly adding to end-to-end latency and consumer lag across the entire topic, not just the partitions that moved. - **Cooperative rebalancing** reduces blast radius but is more complex to implement correctly (it requires two rounds of the protocol) and doesn't eliminate the pause for the partitions that do move. ## What is actually at risk: offset commits The ordering-relevant risk isn't that the log itself gets reordered — offsets and the physical order of records in a partition are completely unaffected by a rebalance; that data is durable and immutable regardless of which consumer is reading it. The risk is at the processing boundary: **offset commits**. A consumer typically processes a batch of records and then commits the offset marking "I've handled up through here." If a rebalance revokes a partition from a consumer after it processed some records but before it committed the offset for them, the new owner resumes from the last committed offset, which is earlier than what was actually processed — so those records get delivered and processed a second time. This is why Kafka's default delivery semantics are "at-least-once": duplicates around rebalance boundaries are expected, and consumers that need correctness under this must either - be naturally idempotent (e.g., upserts keyed by a business ID), - or track processed-record IDs explicitly to detect and skip duplicates. ## Rebalance storms A second, subtler failure mode is a **rebalance storm**: if consumer instances are misconfigured with an aggressive `max.poll.interval.ms` relative to how long each poll's processing actually takes, a slow batch causes the coordinator to declare the consumer dead, triggering a rebalance, which itself adds load and latency, which can cause the next consumer to also miss its window, cascading into repeated rebalances that make almost no forward progress — a classic production incident pattern often diagnosed by seeing heartbeat-failure log lines correlating with lag spikes. ## Where it shows up A concrete real-world example: rolling deployments of a consumer service (say, a Kubernetes rolling update of 6 pods in a group reading a 12-partition topic) intentionally trigger a rebalance per pod restart; teams commonly tune session.timeout.ms/max.poll.interval.ms and adopt cooperative sticky assignment specifically to keep these routine, expected rebalances from causing visible lag spikes or duplicate-processing storms during normal deploys, distinguishing "ordering is preserved because it's a log property" from "processing exactly-once is not, because that's a consumer-offset-commit property."