skip to content

What is the practical difference between map and mapValues (and filter vs transformValues) in Kafka Streams, and why would you prefer mapValues?

level: middleimportance: must knowfreq 65%

answer

  1. map = key+value; mapValues = value only
  2. Key-change sets repartition flag
  3. Flag set even if new key == old key
  4. ValueMapperWithKey reads but can't change key
  5. filter never rekeys

basics

~10 s

map can change both key and value; mapValues changes only the value and keeps the key. Prefer mapValues because keeping the key avoids a repartition topic when a stateful operation follows.

solid answer

~50 s

map takes a `KeyValueMapper` returning a new `KeyValue<K,V>`, so it can rewrite the key. mapValues takes a `ValueMapper` (or `ValueMapperWithKey`, which can READ the key but not change it) and only rewrites the value. The key difference is repartitioning: Kafka Streams conservatively marks a stream as 'requiring repartition' whenever a key-changing operator (map, selectKey, flatMap, transform) is used. If a downstream stateful op (aggregate/join/windowing) follows, Streams injects an internal repartition topic to re-shuffle records by the new key — extra network, an extra topic, and added latency. mapValues never sets that flag, so it's free of repartition cost. Rule of thumb: if you don't need to change the key, use mapValues / flatMapValues / filter rather than map / flatMap / a custom transform. Same logic applies to filter (key-preserving, no flag) — there's no key-changing variant of filter to worry about.

go deeper

for a junior

Know map=key+value, mapValues=value-only.

for a middle

Explain the repartition flag and why mapValues is preferred; know ValueMapperWithKey.

for a senior

Explain that the flag fires even when key is unchanged and only materializes before a stateful op.

for a principal

Design topologies to minimize rekeys; decide when explicit repartition() with controlled partitions beats auto-repartition.

## The APIs - `KStream.map((key, value) -> KeyValue.pair(newKey, newValue))` — a `KeyValueMapper`. It can produce a **new key**. - `KStream.mapValues(value -> newValue)` — a `ValueMapper`. Key is untouched. There's also `mapValues((readOnlyKey, value) -> newValue)` (`ValueMapperWithKey`) which can **read** the key but cannot change it. - `filter((key, value) -> boolean)` / `filterNot(...)` — predicate keep/drop; never changes the key. ## Why the distinction matters: the repartition flag Kafka Streams partitions records by key hash. Stateful operators (aggregations, joins, windowed ops) require **co-partitioning**: every record with a given key must land on the same partition / same task. If you change the key upstream, the existing partitioning no longer matches the new key, so Streams must **re-shuffle**. Streams tracks this with an internal boolean often called the **repartition-required flag**. Any key-*changing* operator — `map`, `selectKey`, `flatMap`, `transform`, `process` (with key change) — sets it. Key-*preserving* operators — `mapValues`, `flatMapValues`, `filter`, `filterNot`, `peek`, `branch`, `merge` — do NOT. The flag does nothing on its own. It only materializes when a **stateful** operator follows: Streams then auto-creates an internal **repartition topic** named like `<application.id>-<name>-repartition`, writes the rekeyed records to it (network round-trip through the broker), and re-reads them so they're correctly partitioned. This adds a topic, broker storage, and latency. Crucial subtlety: the flag is set **even if you set the key to the same value**. `map((k,v) -> KeyValue.pair(k, f(v)))` still flags repartition because Streams cannot prove the key is unchanged. `mapValues(v -> f(v))` does not. That's the concrete reason to prefer mapValues. ## Practical guidance - Need only the value? Use **mapValues** (or `flatMapValues`). - Need to read the key but not change it? Use **mapValues with ValueMapperWithKey**. - Genuinely need a new key? Use **selectKey** (clearer intent than map) or **map**, and accept the repartition when a stateful op follows. - You can call `repartition()` explicitly to control the number of partitions/name rather than letting Streams pick.

  • Does map always create a repartition topic?
    No. map only SETS the repartition-required flag. A repartition topic is created only if a stateful operator (aggregate/join/window) follows downstream. A pure map -> filter -> to(...) pipeline creates no repartition topic.
  • If I write map((k,v) -> KeyValue.pair(k, transform(v))) keeping the same key, is it free?
    No — it still flags repartition because Streams can't prove the key is unchanged. Use mapValues to keep the value-only transform free of repartition cost.

saying these in an interview costs you the question

  • Saying mapValues can change the key
  • Claiming map always creates a repartition topic regardless of downstream
  • Believing setting the new key equal to the old key avoids the flag
  • Thinking ValueMapperWithKey lets you mutate the key

context

resolving links…