On a stream that keeps only the latest value per key, how is a key removed entirely, and why must that record linger?

## Deleting from a store that only appends On a store that keeps only the latest value per key, there is no delete operation in the ordinary sense: the history is appended to, never edited. Removal of a key is expressed the only way anything is expressed here — as a write. The writer appends a **deletion marker**: a record carrying the key and no value. That single mechanism does two jobs at once: - **For the store.** The marker is the newest record for that key, so under keyed retention it supersedes the value, and the next removal pass over that region discards the value. - **For the readers.** Every reader that crosses the marker in the history learns that the key is gone, and drops it from whatever copy of the latest value it maintains. A reader has no other way to find out; an absent record is indistinguishable from a record it has not reached yet. The second job is the one that constrains the design, and it is the reason the marker cannot simply vanish along with the value it replaced. ## Why the marker outlives the value If the marker were removed at the same moment as the value, a reader that had already read the value but had not yet reached the marker's place in the history would never be told. It would keep the key forever, with a value the store has forgotten — and no future pass over the stream would correct it, because there would be nothing left to read. So the marker is given a **grace period**: a bounded span during which it is deliberately kept even though it is itself removable, precisely so that readers still working through the history can cross it. After that, the marker is removed too, and the key leaves the stream without a trace. That makes the grace period a contract, and it is worth stating in one sentence: **the marker's grace period must outlive the slowest full pass over the history that needs to see it.** ## The failure, step by step When the contract is broken, the symptom is a deleted entity coming back to life. The sequence: 1. A reader begins a full pass over the stream — a rebuild of its own copy of the latest value per key — starting at the earliest position still on the store. Early in the pass it reads the current value for key *K*. 2. Meanwhile the writer appends a deletion marker for *K*. 3. The removal pass runs over that region: the old value of *K* is discarded, and the marker remains. 4. The grace period expires. A later pass removes the marker as well. There is now nothing about *K* anywhere in the stream. 5. The slow reader finally reaches that part of the history — and crosses nothing. It never learns about the delete. 6. Its rebuilt state contains *K*, with a value that no longer exists anywhere in the store. The delete has been undone, silently, by a reader. Nothing in that sequence is a bug in the store. Every step did exactly what it promised; the grace period was simply set shorter than the rebuild it had to cover. ## Sizing it, and the alternatives The grace period is not a number to leave at whatever the platform ships. Size it from the real quantity: how long the slowest consumer of this stream takes to work through the whole history, measured on a bad day rather than a good one — a cold rebuild on a loaded cluster, not a warm one. Then leave margin, because rebuilds get slower as the stream grows. | Approach | What it costs | When it is the right call | |---|---|---| | A deletion marker with a grace period sized from the slowest full pass | the operator has to know that number and re-check it as the stream grows | the default; it is the mechanism the store offers | | A value carrying an explicit deleted flag, with no valueless record at all | the key and its value never leave the store, so the floor never comes down | when deletes are rare and readers vary wildly in speed | | Sizing rebuilds down so no pass can outlive the grace period | engineering effort on the reader side, and it must hold as the stream grows | when the stream is shared and you cannot police every reader | Two cautions worth carrying. First, designs genuinely differ: some platforms give the marker a bounded survival of its own, separate from any other bound; others tie it to the general removal schedule; a store that removes a record on acknowledgement has no such concept at all, because there is no standing state to correct. Ask which one you are on rather than assuming. Second, the marker only reaches readers that actually read the stream. Any copy of this data maintained by some other route — a nightly export, a hand-loaded table — is not covered by the marker at all, and that gap is a design problem, not a retention setting.