What does num.standby.replicas do in Kafka Streams, and how does it affect failover and availability?

## The problem standbys solve A **stateful** Streams task (aggregation, join, windowed op) keeps its working state in a local **state store**, typically **RocksDB** on the instance's disk. Every update is also written to a Kafka **changelog topic** (a compacted topic that is the durable source of truth for the store). If the instance hosting that task crashes, the task must be reassigned. Without standbys, the new host has an **empty** store and must **restore** it by replaying the entire changelog from the beginning (or last checkpoint). For a large store this can take minutes — during which the task (and the keys it owns) is unavailable. ## What a standby replica is `num.standby.replicas = N` tells Streams to keep **N additional hot copies** of each stateful task's store on **other** instances. A standby: - Does **not** process input or produce output. - **Continuously consumes the changelog topic** for that task, applying updates to its own local RocksDB copy. - Stays **nearly in sync** with the active task's store. ## Failover behavior When the active instance fails, the partition assignor prefers to **promote a standby** that already holds the state. Because the standby is almost caught up, the new active task only needs to consume the **small tail** of changelog records produced since the standby last polled — recovery goes from minutes to (often) sub-second to seconds. This is the 'hot failover' the coverage area refers to. ## Interaction with warmup replicas (KIP-441) Distinct concept: **warmup replicas** are temporary copies created during **scaling/rebalancing** to pre-restore a task that is about to move, so the move doesn't stall processing. Standby replicas are **permanent** copies for **failover**. They share the same restoration machinery but serve different goals; `max.warmup.replicas` caps concurrent warmups. ## Costs and constraints - **Resources**: each standby costs disk + memory (another RocksDB instance) and network/CPU to follow the changelog. N=1 roughly doubles state footprint. - **Placement requirement**: you need enough distinct instances. With N=1 you need ≥2 instances for the standby to be placed on a *different* host; otherwise it can't help with machine failure. - **Not throughput**: standbys never run active tasks, so they add zero processing capacity — purely availability/recovery. - **Rack awareness** (KIP-708): `rack.aware.assignment.tags` can force standbys onto different racks/AZs so an AZ outage still leaves a usable copy. ## Edge cases - A standby can lag if it can't keep up with changelog throughput; on promotion it still replays the lag tail, so recovery isn't strictly instant. - Stateless tasks have no state store and thus no standby cost — standbys only matter for stateful sub-topologies. - Combined with `acceptable.recovery.lag`, the assignor decides whether a standby is 'caught up enough' to be promoted directly.