How does cooperative rebalancing work in Kafka Streams, and why was it introduced over the older eager protocol?

## Background: what a rebalance is A **rebalance** is the Kafka consumer-group process of (re)assigning partitions/tasks to members when membership changes (an instance joins, leaves, or crashes) or topic metadata changes. Kafka Streams piggybacks task assignment on the consumer group protocol via the **StreamsPartitionAssignor**. ## The old way: eager rebalancing Under the **eager** protocol, the rebalance is **stop-the-world**: 1. Every member **revokes ALL** of its partitions (stops processing entirely). 2. The leader computes a brand-new assignment. 3. Every member gets its (possibly different) partitions and resumes. Problems: - **Global pause**: all processing halts during the rebalance, even for tasks that won't move. - **State restoration cost**: a stateful task is backed by a local **state store** (RocksDB) whose updates are also written to a **changelog topic**. If the task moves to a new instance, that instance must **restore** the store by replaying the changelog from the start (or from a checkpoint) before it can process — potentially minutes of downtime for large stores. ## The new way: cooperative (incremental) rebalancing — KIP-429 Default since Kafka 2.4. The key idea: **only revoke what actually moves**, in two passes. 1. **First rebalance**: the leader computes the desired final assignment. Members compare it to what they currently own and **revoke only the tasks being reassigned away**. Crucially, they **keep processing all tasks they get to retain** — no global pause. 2. **Second (follow-up) rebalance**: the now-free partitions are handed to their new owners, who begin restoring/processing them. The assignor is **sticky**: it tries to keep each stateful task on the instance that already has its local state, so it avoids changelog restoration whenever possible. ## Why it matters for scaling - **Rolling restarts / deploys**: bouncing one instance only moves that instance's tasks; the rest of the fleet keeps running. - **Incremental scale-out**: adding one instance moves just enough tasks to balance, instead of churning the whole assignment. - **Warmup replicas (KIP-441)**: builds on cooperative rebalancing — when a task must move, Streams first spins up a **warmup replica** that restores state in the background, and only transfers the active task once it's caught up (governed by `acceptable.recovery.lag` and probing rebalances every `probing.rebalance.interval.ms`, default 10 min). This keeps availability high during scaling. ## Edge cases & gotchas - **Upgrades**: moving from eager to cooperative requires a careful two-rolling-bounce upgrade path (set `upgrade.from` on the first bounce) because the two protocols can't be mixed naively. - A misbehaving member that fails to rejoin within `session.timeout.ms` / `max.poll.interval.ms` still triggers reassignment of its tasks. - Cooperative rebalancing reduces but does not eliminate restoration: a task genuinely moving to a fresh instance still restores (warmup replicas hide most of that latency).