A GDPR erasure request requires deleting all of a user's data from a compacted Kafka topic. How do tombstones and retention/compaction make this work, and what are the caveats?

## The problem GDPR's right to erasure means you must be able to delete a person's data. Kafka logs are **append-only and immutable** — you cannot edit or delete an individual record in place. Two mechanisms make erasure possible: **retention** and **compaction with tombstones**. ## Retention vs compaction Every topic has a `cleanup.policy`: - **`delete`** (default): records are removed once they exceed **`retention.ms`** (age) or **`retention.bytes`** (size). This ages out *all* old data on a schedule but **cannot target a single user**. - **`compact`**: Kafka keeps **at least the latest value for each message key**; older values for the same key are eligible for removal. Used for changelog/state topics where you want the current state per key. ## Tombstones In a **compacted** topic, a **tombstone** is a record with a **non-null key and a `null` value**. It means 'this key is deleted.' During the next compaction pass: 1. All earlier records for that key are removed. 2. The tombstone is retained for a grace window — **`delete.retention.ms`** (default 24h) — so consumers that are behind still observe the deletion. 3. After that window, the tombstone itself is purged. So to erase a user from a compacted topic keyed by user id: **produce a tombstone with their key**. ## Why deletion is not immediate Compaction is a **background process** with knobs: - It only acts on **non-active (closed) segments** — the segment currently being written is never compacted, so very recent data lingers until the segment rolls (`segment.ms` / `segment.bytes`). - The **log cleaner** triggers based on **`min.cleanable.dirty.ratio`** (how much of the log is 'dirty'/uncompacted) and **`min.compaction.lag.ms`** / **`max.compaction.lag.ms`** (minimum/maximum age before a record is compacted). Result: erasure is **eventually consistent**. You generally **cannot promise deletion within seconds**; you can tune `max.compaction.lag.ms` to bound the worst case. ## Caveats and gaps - **Keyed-by-subject required:** tombstones erase **per key**. If a user's data is scattered under many keys, embedded in events keyed by something else, or in multiple topics, a single tombstone won't reach it — you need a per-topic deletion design. - **Downstream propagation:** consumers, **Streams changelog/state stores**, materialized views, search indexes, data lakes, and **tiered storage** must also process the tombstone or honor the deletion — otherwise copies survive. - **Compaction must be on:** a `delete`-policy topic ignores tombstones for targeted erasure; tombstones only have erasure semantics under `compact`. - **Crypto-shredding complement:** when guaranteed physical deletion is hard (backups, tiered/object storage, replicas), encrypt each subject's data with a per-subject key and, on erasure, **destroy the key** — the ciphertext becomes permanently unreadable, satisfying erasure without chasing every byte. ## Summary Tombstone + compaction = targeted, eventual, per-key deletion. Retention = bulk, time-based aging. Crypto-shredding = make-unreadable when delete-everywhere isn't feasible.