Where does stateful windowing state live in Kafka Streams versus Flink versus Spark, and how is it made fault-tolerant?

## Why state needs special handling Stateless transforms (map, filter) can restart anywhere. **Stateful** ones — counting per key, windowed aggregates, joins, deduplication — accumulate data that must survive crashes and must move correctly when work is rebalanced. Each engine pairs **fast local state** with a **durable backup**. ## Kafka Streams - **Local store**: each task holds an embedded **RocksDB** instance on local disk (in-memory stores are also available for small state). Lookups and updates are local and fast. - **Durability**: every write to a state store is also appended to a **changelog topic** in Kafka, which is **log-compacted** so it retains the latest value per key. The changelog is the source of truth. - **Recovery / rebalance**: when an instance dies or partitions move, the receiving instance **restores** the store by consuming its changelog from the beginning (compacted, so bounded by key count). To avoid slow restores, configure **standby replicas** (`num.standby.replicas`) that keep warm copies; Streams can also use **interactive queries** to read local state directly. - **Operational note**: state is tied to disk on the app instances. Losing a node means restoring from changelog; large state means long restores without standbys. ## Apache Flink - **State backends**: `HashMapStateBackend` keeps state as objects **on the JVM heap** (fast, limited by memory); `EmbeddedRocksDBStateBackend` keeps it **off-heap in RocksDB** on local disk, enabling **very large state** and **incremental checkpoints** (only changed RocksDB SST files are uploaded). - **Checkpointing**: Flink periodically runs **asynchronous barrier snapshots** — barriers flow through the dataflow and each operator snapshots its state to **durable storage** (S3, HDFS, etc.). Checkpoints are for automatic failure recovery. - **Savepoints**: user-triggered, portable snapshots used for upgrades, rescaling, and code changes. They are the operational lever for stateful redeploys. - **Recovery**: on failure Flink restores all operator state from the last completed checkpoint and rewinds sources to the matching offsets. ## Spark Structured Streaming - **Checkpoint location**: a directory on durable storage containing the **offset log** (which input ranges per batch), the **commit log**, and the **state store** data. - **State store**: versioned per batch; backed by HDFS-compatible storage by default, with an optional **RocksDB state store** for large state to reduce JVM GC pressure. - **Recovery**: on restart Spark reads the checkpoint, identifies the last committed batch, and resumes; state is restored from the versioned store. Watermarks bound how much windowed/join state is retained. ## Comparison | Aspect | Kafka Streams | Flink | Spark | |---|---|---|---| | Local state | RocksDB / in-memory | heap or RocksDB | heap or RocksDB | | Durable backup | compacted changelog topic (in Kafka) | checkpoints to S3/HDFS | checkpoint dir (offset+state) | | Incremental backup | n/a (changelog is per-write) | yes (RocksDB incremental) | versioned state files | | Manual snapshot for upgrades | n/a (changelog replay) | savepoints | restart from checkpoint | ## Edge cases - **Restore time**: huge Streams state without standbys means long rebalance pauses; size standbys vs cost. - **Checkpoint storage failures** (Flink/Spark) can stall the job; the durable store is a hard dependency. - **Schema/topology changes**: Streams changelog and Flink savepoints both have compatibility rules; changing key/serde or operator UIDs can break restore.