A team turns on keyed retention to shrink a stream before a deadline — what does keeping only the latest value per key actually promise?

## Two rules, two different promises A store holding a stream can be told to remove records in one of two ways. **Age-based removal** (oldest-first) says anything older than a bound may go. **Keyed retention** — keeping only the latest value per key — says something structurally different: any record for which a newer record carrying the same key exists may go. Not every platform in this class offers the second rule. Designs that remove a record once a subscriber acknowledges it have no notion of it, and some rented brokers expose only the first. Where it is offered, the guarantee runs in exactly one direction: **for every key that has been written and not explicitly deleted, the most recent value stays readable**. A reader that begins at the earliest position still on the store and reads to the end finishes holding the current value of every such key. That is the whole contract, and it is a property about survival. Three things it pointedly does not promise: - **When** a superseded value disappears. It becomes *eligible* for removal, and nothing states a deadline. - That a superseded value is unreadable in the meantime. A reader crossing old history may cross several earlier values for the same key before reaching the current one. - That the stream reaches, or stays under, any particular size. ## Why it is not a capacity lever The removal that keyed retention authorises is carried out by a background **removal pass**. The pass reads closed segments, keeps the surviving record for each key, writes the result into new segments and then releases the old ones. It never touches the **active segment** — the one still being appended to — so the newest slice of the stream always holds every value written into it, superseded or not. Platforms differ in how they decide when to run it: some schedule by elapsed time, some by how much of the eligible region is worth rewriting, and some expose no control at all. None of them offers what a capacity plan needs, which is *this stream will be under a stated number of bytes by Friday*. | What an operator wants to know | Keyed retention | Age-based removal | |---|---|---| | What is guaranteed to survive? | the latest value of every live key | everything newer than the bound | | What is guaranteed to be gone, and by when? | nothing, at no stated time | everything past the bound, once the pass runs | | Is the resulting size predictable? | only from key count and value size | roughly, from write rate times the bound | | Does it bound how far back history goes? | no | yes | ## What actually sets the floor A keyed stream's floor is **the number of distinct keys, times the size of a value, times the number of copies the cluster keeps** — plus the superseded bytes the pass has not reclaimed yet, plus the active segment. That has consequences worth saying out loud: - If keys are never reused — a fresh identifier on every record — nothing is ever superseded, and keyed retention reclaims nothing however long you wait. - If values are large, the floor is large even at modest key counts, because every live key contributes one value. - If the write rate is high relative to how fast the pass rewrites, the unreclaimed remainder is not a rounding error and can dominate the total. - Switching the rule on moves nothing by itself: no bytes are released until a pass has run over the closed segments. ## When the deadline is real If a store has to fit a number by a date, the instruments are **an age bound** (the maximum age a record may reach before it becomes removable) and **a byte ceiling** (the maximum bytes a stated scope may occupy), because each is enforced against a quantity you chose. Keyed retention answers a different question: what must a reader be able to reconstruct, and in how much work. The two questions are independent, which is why some platforms let both rules apply to one stream — and that combination removes more than people expect, because a record past the age bound goes even when it is the latest value for its key. It is worth being explicit about what is being traded. The retention window — the span of history still readable on the store right now — **is the replay budget**: it is the amount of history anything downstream can be rebuilt from. Keyed retention deliberately gives that budget up for superseded values, in exchange for a bounded path to current state. Choosing it is a statement about the stream's contract with its readers, not a knob to reach for when space is short.