Why does the pass enforcing keyed retention need free space on the store, and how can it fall behind?

## Truncating against rewriting Both retention rules end with bytes being released, but the mechanism underneath them is not comparable. **Age-based removal** (oldest-first) is a truncation. A **segment** — the unit the store rolls closed and removes as a whole — is either entirely past the bound or it is not. When it is, the store releases it. No record is inspected, nothing is copied, and the space comes back at once. **Keyed retention** — keeping only the latest value per key — cannot work that way, because the records that must survive are scattered through the same segments as the records that may go. So the **removal pass** rewrites instead: it reads a run of closed segments, decides per key which record is the survivor, writes the survivors into fresh segments, makes the new segments the readable ones, and only then releases the originals. | | Age-based removal | Keyed retention | |---|---|---| | Unit of work | a closed segment, released whole | a run of closed segments, read and rewritten | | Cost | near zero — nothing is read | proportional to the bytes scanned, not to the bytes reclaimed | | Free space needed | none; it frees space | yes — the survivors exist twice until the swap completes | | The active segment | never removed | never rewritten | | Can it fall behind? | rarely; the work is trivial | yes, whenever writes outrun the rewrite rate | ## Why headroom is a precondition The pass writes before it releases, because a crash in the middle must not lose the survivors. For the duration of a pass, the surviving records therefore exist in two places at once, and the store's usage peaks above its steady state. Two practical consequences follow: - A store that has been run to the edge of its data volume can be *unable to reclaim*: there is no room to stage the rewrite, so the pass cannot complete, so nothing is released. The rule that was supposed to bound the stream stops working precisely when it is needed. - Capacity planning for a keyed stream has to budget for the peak, not the floor. How large the peak is depends on how much the pass takes on at a time, and platforms differ: some rewrite a small run of segments per cycle, some much more. The same ordering is why a pass is safe to interrupt. If it dies before the swap, the original segments are still the readable ones and the work is simply repeated. ## The removal pass's own lag The pass is background work, scheduled against a live write path that is using the same device and the same processors. Its **own lag** — how far behind the eligible region it is — is the number that matters operationally, and it is not the same thing as the distance between a reader and the newest record, which is a different measurement entirely. What makes it grow: 1. **Churn.** If the same keys are rewritten fast, superseded bytes accumulate faster than the pass converts them back into free space. The footprint climbs even though the number of live keys is constant. 2. **Scheduling policy.** Platforms decide differently when a region is worth rewriting: some by elapsed time, some by how much of the region is superseded, some not at all in any way you can influence. A low-churn stream may sit unrewritten for a long time and that is normal, not a fault. 3. **Competition.** The pass yields to serving writes and reads. On a saturated device it makes little progress at exactly the moment its output is most wanted. 4. **One very large stream.** Work is usually taken a stream or a partition at a time, so a single enormous one can hold the pass while everything else waits. The operational readings that follow from this are worth stating plainly: - A keyed stream's size climbing is not evidence the rule is off; it is usually evidence the pass is behind. - The delay before a superseded value actually becomes unreadable is a *consequence* of the pass's progress, not a setting. Anything downstream that depends on old values being gone by a deadline is depending on something the store never promised. - Giving the pass more parallelism, where a platform exposes that, is the direct remedy; reducing churn or value size is the indirect one. ## What an operator watches Keep free space on the data volume deliberately — enough to stage a rewrite, not merely enough to accept the next write. Watch the removal pass's own lag and the ratio of stream size to the expected live-key footprint; a growing gap between them is the early signal. And treat a stream whose keys never repeat as a stream the pass can do nothing for, however long it runs.