skip to content

What happens when a serializer is given a null value, and what is a tombstone?

level: middleimportance: must knowfreq 58%

answer

  1. serialize(null) -> null bytes, no crash
  2. tombstone = non-null key + null value
  3. only meaningful on cleanup.policy=compact
  4. delete.retention.ms default 24h
  5. KTable null value = delete key

basics

~20 s

Built-in serializers return null bytes for null input — they don't crash. A record with a non-null key but a null value is a tombstone: on a log-compacted topic it signals 'delete this key,' and compaction eventually removes it.

solid answer

~50 s

Kafka allows a record's key and/or value to be null, and the message itself carries a null/not-null flag separate from any byte content. The built-in serializers (String, Long, ByteArray, etc.) all return null from serialize() when handed null, so producing a null value is legal. The important case is a record with a non-null key and a null value on a topic configured with cleanup.policy=compact: this is a tombstone. Log compaction interprets it as 'the latest known state for this key is deleted,' so downstream consumers and compacted state see the key removed. The tombstone is retained for at least delete.retention.ms (default 24h) so consumers can observe the delete, then it is purged. On the consumer side, the deserializer is also handed null and returns null — so application code must null-check. Kafka Streams KTables treat a null value as a delete of that key.

go deeper

for a junior

Know that null values are allowed and that a null value with a key can mean 'delete'.

for a middle

Define a tombstone precisely and tie it to log compaction and consumer null-handling.

for a senior

Reason about delete.retention.ms, null-key edge cases, and Streams KTable delete semantics.

for a principal

Design CDC/state-store deletion strategies and educate teams on tombstone retention windows and compaction interplay.

## Null is a first-class concept In Kafka, a record's key and value can each independently be `null`. This is not the string "null" or empty bytes — the record format stores a length of -1 to mean 'absent.' So 'null value' is distinct from 'zero-length value.' ## Serializers and null Every built-in serializer follows the same contract: `serialize(topic, null)` returns `null` (not an empty array, not an exception). So you can freely produce a record whose value is null. The matching deserializer does the inverse: handed `null` bytes, it returns `null` to your application. Custom serializers should follow this convention. ## What a tombstone is A **tombstone** is a record with a **non-null key** and a **null value**. It only has special meaning on a **log-compacted** topic (`cleanup.policy=compact`). Log compaction keeps, per key, at least the latest value. A tombstone tells compaction: 'the latest value for this key is deletion' — so after compaction the key effectively disappears from the compacted view. ## Retention of tombstones A tombstone is not removed immediately, or consumers that are behind might never see the delete. It is retained for at least `delete.retention.ms` (**default 86400000 ms = 24h**) after the segment becomes eligible for compaction. After that window, compaction removes the tombstone itself, reclaiming space. ## Streams semantics In Kafka Streams, a `KTable` is a changelog: a record with key K and a non-null value is an upsert; a record with key K and a **null** value is a **delete** of K. This is exactly the tombstone semantics and is why `KStream.toTable()`/aggregations emit nulls to delete. ## Common pitfalls - Forgetting to null-check after deserialization → `NullPointerException` in the consumer. - Producing a null **key** with a null value on a compacted topic: a null key is not a tombstone target (compaction needs a key); brokers may even reject null-keyed records on compacted topics. - Confusing an empty `byte[0]` with a tombstone — only a true `null` value is a tombstone. - Assuming a tombstone deletes instantly; the `delete.retention.ms` window keeps it visible for late consumers.

  • Why isn't a tombstone deleted from the log immediately?
    It must survive delete.retention.ms (default 24h) so lagging consumers can observe the delete before compaction purges the tombstone itself.
  • Is an empty byte array the same as a tombstone?
    No. A tombstone is a true null value (length -1 on the wire). A zero-length byte[] is a present, empty value and does not trigger compaction deletion.

saying these in an interview costs you the question

  • Saying serializers throw on null input — built-ins return null.
  • Calling any null-value record a tombstone regardless of topic config — it only matters under cleanup.policy=compact.
  • Confusing byte[0] (empty present value) with null (tombstone).
  • Claiming the tombstone is purged the instant compaction runs.

context

resolving links…