State Stores and Fault Tolerance

## What problem state stores solve Kafka Streams processes an unbounded stream of records (key-value pairs read from Kafka topics). Some operations are **stateless** — `filter`, `map`, `flatMap` — each output depends only on the current record. Others are **stateful** — `count`, `aggregate`, `reduce`, windowing, and joins — where the output for a record depends on records seen *before* it. To compute `count()` per key, the app must remember the running count for each key; that memory is the **state store**. ## What a state store is, concretely A state store is a local, embedded key-value database that lives **inside the Kafka Streams process** on each application instance. It is not a remote database — there is no network call to read or write it, which is what makes stateful operations fast and scalable. The default implementation is **RocksDB**, an embedded log-structured key-value store written in C++ that persists to local disk. There is also an **in-memory** implementation (a `Map` on the JVM heap). ## Partitioning: state is sharded with the data Kafka Streams divides work into **tasks**, one per input-topic partition (per sub-topology). Each task owns its own slice of state — the state for the keys in *its* partition. When you run multiple instances, tasks (and therefore state shards) are distributed across them. So no single instance holds all the state; each holds only the state for the partitions it currently processes. This is how state scales horizontally. ## Where state lives on disk Persistent (RocksDB) stores are written under the directory configured by `state.dir` (default `/tmp/kafka-streams`), in a subdirectory named after the `application.id`. In production you should set `state.dir` to durable, fast local storage (not `/tmp`, which can be cleared on reboot). ## Fault tolerance is separate Local disk can be lost (instance crashes, container is rescheduled). So a state store on its own is not durable. Kafka Streams makes it fault-tolerant by backing each store with a **changelog topic** in Kafka — every update to the store is also written to that topic, so the store can be reconstructed elsewhere. That is the key insight: the *local store* is the fast read path; the *changelog topic* is the durable source of truth. ## When you don't have a store A pure stateless pipeline (read → filter → map → write) has no state store at all, and therefore nothing to restore and no changelog. State stores appear only when you introduce stateful DSL operators or explicitly add a store in the Processor API.