What is the difference between a KStream and a KTable in the Kafka Streams DSL, and when would you use each?
answer
- KStream = events (append)
- KTable = upserts (latest per key)
- null in KTable = tombstone/delete
- stream-table duality
- KTable materialized + compacted changelog
basics
~20 sA KStream is a record stream where every record is an independent event (insert/append). A KTable is a changelog stream where each record is an update keyed by its key — only the latest value per key matters.
solid answer
~50 sA KStream models an unbounded sequence of independent events: every record is appended, and two records with the same key are two distinct facts (e.g. 'page-view', 'click'). A KTable models a changelog: each record is an UPSERT for its key, so the table holds only the latest value per key, and a null value is a tombstone (delete). Use a KStream for event/transaction streams where every occurrence matters (logs, clicks, payments). Use a KTable for mutable entity state where you only care about the current value (user profile, account balance, inventory level). Under the hood a KTable is materialized into a local state store backed by a compacted changelog topic, which is what enables lookups and joins against current state. The DSL lets you convert between them: stream().toTable() / aggregate() turns a stream into a table, and table.toStream() emits the changelog as events.
go deeper
Recall the one-liner: KStream = events, KTable = latest value per key; null deletes in a KTable.
Explain stream-table duality and the conversions (toTable/aggregate/toStream) and when each fits.
Discuss materialization into a RocksDB state store + compacted changelog, tombstone semantics, and cache/commit-interval compaction of updates.
Reason about modeling choices for a whole pipeline: which abstraction minimizes repartitions and state, and the correctness implications of tombstones and update compaction downstream.
## First principles Kafka Streams gives you three core abstractions built on top of Kafka topics. Two of them — **KStream** and **KTable** — express the famous **stream-table duality**: a stream of events and a table of state are two views of the same data. ### KStream — a record (event) stream A `KStream<K, V>` represents an **unbounded, append-only sequence of records**. Each record is an **independent, immutable fact**. If two records arrive with the same key, they are two separate events, NOT an update of one another. Example: a topic of page views ``` ("alice", view-home) , ("alice", view-cart) ``` These are two distinct page views by Alice — you keep both. ### KTable — a changelog (update) stream A `KTable<K, V>` represents an **evolving table** where each record is an **UPSERT** (insert-or-update) for its key. Only the **latest value per key** is meaningful. A record with a **null value is a tombstone** that deletes the key. Example: a topic of account balances ``` ("alice", 100) , ("alice", 80) ``` The table now holds `alice -> 80`; the 100 is overwritten. A KTable is **materialized** into a local **state store** (RocksDB by default), backed by a **compacted changelog topic** for fault tolerance. ### Stream-table duality - A **stream is a changelog of a table**: replay the changelog and you rebuild the table. - A **table is a snapshot of a stream**: take the latest value per key and you get the table. The DSL exposes conversions: - `KStream.toTable()` or aggregations (`groupByKey().reduce()/aggregate()/count()`) → KTable. - `KTable.toStream()` → emits the table's changelog as a KStream of events. ### When to use which - **KStream**: clickstreams, transactions, IoT readings, audit logs — anything where every occurrence is a fact you must process. - **KTable**: current entity state — user profiles, configuration, inventory, balances — where you only care about the latest value and want efficient key lookups/joins. ### Edge cases - **Null semantics differ**: in a KStream a null value is just a record (and may be filtered); in a KTable a null value **deletes** the key (tombstone). - KTable updates are **not** guaranteed to emit one downstream record per input — record caching and `commit.interval.ms` can **compact** intermediate updates so you only see the last value within a window of time (this is the suppression/cache behavior, not a bug). - A KTable needs a **keyed** source; if your data isn't keyed by the entity, you must `groupBy`/`selectKey` and aggregate, which triggers a repartition.
- What does a null value mean in a KStream versus a KTable?In a KStream a null value is just another record (you may filter it). In a KTable a null value is a tombstone that deletes the key from the table and its state store/changelog.
- How do you convert a KStream into a KTable?Either KStream.toTable() (treat the stream directly as a changelog) or a stateful aggregation: stream.groupByKey().reduce()/aggregate()/count(), which produces a KTable. groupBy with a new key triggers a repartition first.
saying these in an interview costs you the question
- Saying a KTable keeps all records like a KStream — it only keeps the latest value per key.
- Claiming KStream supports key-based lookups/joins-against-state — that's the KTable's materialized store.
- Forgetting that a null value deletes a key in a KTable (treating it as a normal record).