Walk through what happens to state stores during a rebalance, and explain how standby replicas and state.dir affect restoration time.

## Why rebalances threaten state Kafka Streams instances form a consumer group. When membership changes — an instance joins, leaves, or crashes, or partitions are added — the group **rebalances** and tasks are reassigned. A **stateful task** carries a state store. If the task moves to an instance that doesn't already have that store on local disk, the store must be **restored** before processing can resume, because correctness requires the full prior state (you can't resume a running count from empty). ## The restoration sequence 1. After assignment, a task with missing/stale local state goes into the **RESTORING** state. 2. A dedicated **restore consumer** reads the store's **changelog** partition from the last checkpointed offset (or from the beginning if there's no local state) up to the high-water mark. 3. Each changelog record is written back into the local store (RocksDB or in-memory), rebuilding it. 4. Only when restore completes does the task become **RUNNING** and start processing new input. `restore.consumer.*` and `max.poll.records` influence throughput here; the `StateRestoreListener` lets you observe progress. Restoration time is **proportional to changelog size** (number of records to replay). For a large store this can mean minutes of unavailability for the affected partitions — a real production concern. ## Mitigation 1: standby replicas Setting **`num.standby.replicas=N`** tells Streams to maintain *N* extra warm copies of each store on other instances. A standby task continuously tails the changelog into a local store but does no processing. When the active instance fails, the coordinator promotes a standby whose state is already (nearly) current, so restore is near-instant instead of a full replay. Standbys cost extra disk, memory, and changelog read bandwidth, and add a small replication lag window, but they are the primary lever for fast failover and high availability. ## Mitigation 2: durable state.dir and stickiness If **`state.dir`** points at storage that **survives instance restarts** (e.g., a mounted volume, not ephemeral `/tmp`), a restarting instance finds its RocksDB files and a **checkpoint file** recording how far the store was restored. It then replays only the **tail** of the changelog (records after the checkpoint), which is fast. The **cooperative-sticky** assignment strategy and **probing rebalances** (KIP-441, Streams 2.6+) actively try to keep/return each task on the instance that already holds its state, avoiding cold restores. Streams reports a per-task **lag** so the assignor can move work to the warmest instance. ## Exactly-once interaction Under **exactly-once (EOS)**, a crash may have left uncommitted writes in the local store. On recovery Streams uses the checkpoint to discard state past the last committed offset and restores forward from the changelog, so aborted/duplicate writes don't survive. This is why EOS may wipe and rebuild local state more aggressively than at-least-once. ## Putting it together (capacity/SLA view) - Big stores + no standbys + ephemeral state.dir = worst case: full changelog replay on every failover. - Standbys + durable state.dir + cooperative-sticky = near-zero restore in the common case. - The trade-off is resource cost (standby disk/mem/bandwidth) vs. recovery-time objective.