Explain the difference between eager (stop-the-world) and incremental cooperative rebalancing (KIP-429).
answer
- eager = revoke ALL, stop-the-world
- cooperative = revoke only movers, keep processing
- two rebalances: compute+revoke, then assign freed
- KIP-429; CooperativeSticky + Kafka Streams
- migrate: list both, then drop eager
basics
~20 sIn eager rebalancing every consumer revokes all its partitions and stops processing until a new assignment arrives (stop-the-world). Incremental cooperative rebalancing (KIP-429) only revokes the partitions that actually need to move, so most partitions keep being processed throughout.
solid answer
~50 sEager rebalancing is the classic protocol: when a rebalance starts, each member sends JoinGroup and revokes ALL of its partitions, halting consumption; the leader computes a fresh assignment and everyone resumes via SyncGroup. Throughput drops to zero group-wide during the gap. KIP-429 introduced incremental cooperative rebalancing, used by CooperativeStickyAssignor and Kafka Streams. It splits the work into TWO rebalances: the first computes the target assignment but only revokes partitions that must change owners (members keep everything else and keep processing); a second, triggered rebalance then assigns those now-free partitions to their new owners. The cost is an extra round-trip, but the benefit is that the vast majority of partitions never stop. This is what makes rolling restarts and autoscaling cheap. Migration from eager to cooperative must be done by listing both assignors during a rolling upgrade.
go deeper
Know that eager stops everything during a rebalance while cooperative keeps most partitions running.
Describe the two-phase revoke/assign flow and that CooperativeStickyAssignor + Kafka Streams use it.
Explain the single-owner safety reason for two rebalances and execute the two-step eager->cooperative migration correctly.
Weigh the availability-vs-extra-latency trade-off across a fleet, design rolling-deploy strategies, and reason about callback semantics (revoked/assigned/lost) under each protocol.
## The problem with eager Under the original **eager** protocol, the rebalance lifecycle is: a trigger fires -> the coordinator tells all members to rejoin -> in `onPartitionsRevoked` each member **gives up every partition it owns** and stops calling `poll()` for data -> the leader runs the assignor -> members receive the new assignment in `onPartitionsAssigned` and resume. Because *all* partitions are revoked up front, the **entire group stops processing** for the duration — a 'stop-the-world' pause. In a large group, or during a rolling deploy where each restarting pod triggers a rebalance, these pauses stack up and tank throughput / inflate latency. ## Incremental cooperative rebalancing (KIP-429) The insight: most partitions don't need to move. If C1 keeps p0 and only p1 needs to migrate from C1 to a new consumer C3, why stop C1 from processing p0? Cooperative rebalancing implements this with a **two-phase, revoke/assign** flow: 1. **Rebalance 1 (compute + revoke-only):** Every member rejoins and reports what it currently owns. The cooperative assignor computes the desired end-state but, crucially, returns to each member only its *retained* partitions; partitions that must change hands are **revoked by their current owner** (and only those). Members keep processing everything they retained. No global stop. 2. **Rebalance 2 (assign the freed partitions):** Revoking partitions triggers a second rebalance. Now the partitions that were freed in phase 1 are *safely* assigned to their new owners, because the old owner has provably released them. This preserves the invariant 'one owner per partition' without ever double-assigning. So a single logical reassignment becomes **two rebalances**, but only the moving partitions ever pause. ## Where it is used - `CooperativeStickyAssignor` for plain consumers. - Kafka **Streams** uses cooperative rebalancing by default (its `StreamsPartitionAssignor`). ## Protocol detail The consumer advertises a `protocol` (EAGER or COOPERATIVE) in its JoinGroup metadata. All members in a generation must agree. The `RebalanceProtocol` is derived from the assignor: `CooperativeStickyAssignor.supportedProtocols()` includes COOPERATIVE. ## Migration (important, easy to get wrong) You cannot flip a running group from eager to cooperative in one step. The safe path (per KIP-429): 1. Roll out a version that lists BOTH `[CooperativeStickyAssignor, StickyAssignor]` (or whatever pair). While any member still only supports eager, the group uses eager. 2. Once **every** member supports cooperative, do a second roll that lists ONLY `[CooperativeStickyAssignor]`. Now the group uses cooperative. Skipping the two-step can corrupt the assignment because eager and cooperative make different assumptions about what is revoked. ## Edge cases / trade-offs - Cooperative trades latency (extra rebalance round-trip) for availability (no global stall). For tiny groups the eager pause may be negligible. - Callbacks differ: with cooperative, `onPartitionsRevoked` receives only the partitions actually being lost, and `onPartitionsAssigned` only the newly gained ones — `onPartitionsLost` handles the abnormal case (e.g. fenced) where you didn't get to commit. - It does NOT eliminate rebalances; it makes each one cheaper.
- Why does cooperative rebalancing need two rebalances instead of one?To preserve the single-owner invariant safely. In rebalance 1 the current owners revoke the partitions that must move; only after they've provably released them does rebalance 2 assign those freed partitions to new owners. Doing it in one pass would risk two consumers briefly believing they own the same partition.
- How do you migrate a live consumer group from StickyAssignor (eager) to CooperativeStickyAssignor without downtime?A two-phase rolling upgrade. First deploy with partition.assignment.strategy listing BOTH cooperative and the old assignor; the group stays eager until everyone supports cooperative. Once all members are upgraded, deploy again with only CooperativeStickyAssignor so the group switches to the cooperative protocol.
- Does cooperative rebalancing reduce the NUMBER of rebalances?No — it can actually double the rebalances per logical change (the two-phase flow). What it reduces is the COST of each rebalance: only the partitions that move stop processing, instead of the whole group going stop-the-world.
saying these in an interview costs you the question
- Saying cooperative rebalancing eliminates rebalances (it makes them cheaper, and uses two per change).
- Claiming you can switch eager->cooperative in a single deploy without the two-phase migration.
- Thinking eager only pauses the affected consumer (it pauses the whole group).
- Confusing 'sticky' (minimizes movement in the final assignment) with 'cooperative' (avoids revoking partitions that aren't moving).