For a stream carrying current state per key, do you run keyed retention, age-based removal, or both — and what does both actually remove?

## Ask what the stream is, not how big it is The rule that governs removal is a statement about the contract a stream offers its readers. Two clean cases and one mixed one: - **A record of things that happened.** Each record is independently meaningful — a payment, a reading, a state transition. No record supersedes another, so keyed retention would have nothing to work with even if you enabled it. The rule is **age-based removal** (oldest-first), and the bound is a budget decision, because the retention window — the span of history still readable — **is the replay budget**: the amount of history anything downstream can be rebuilt from, and the amount of time a stalled reader may stay stalled. - **The current value of a set of keys.** The stream carries, for each key, its latest state, republished whenever it changes. What a new reader needs is not the history but the current picture, reached in work proportional to the number of keys rather than to the age of the stream. That is exactly what **keyed retention** provides. - **Short-lived state per key.** Keys exist for a while — a session, an order in flight, a device that reports for a season — and then stop being written. Neither rule alone fits: keyed retention keeps every key that ever existed forever, and age-based removal alone keeps every superseded value inside the bound. ## What 'both' actually removes This is the part that is routinely misread. Where a platform permits both rules on one stream, they are not two filters that must agree before anything goes. They are two independent authorisations to remove: - keyed retention authorises removing a record **because a newer record with the same key exists**; - the age bound authorises removing a record **because it is older than the bound** — and that authorisation does not care whether the record is the latest value for its key. So the combination does **not** mean *keep the latest value per key, and additionally trim stale history*. It means *keep the latest value per key, unless that value is old, in which case remove it too*. A key that stops being written and then ages past the bound loses its latest value, and once that has happened the key is gone from the store entirely: a reader rebuilding from the earliest position still on the store will never learn it existed. | The stream is… | Rule | What a new reader gets | The trap | |---|---|---|---| | independently meaningful history | age-based removal | everything inside the retention window | the window is the replay budget; size it from how long a reader may be down | | current state per key, keys live indefinitely | keyed retention | the latest value of every key | the footprint floor is key count times value size times copies | | current state per key, keys go quiet and die | both | the latest value of every key still being written | quiet keys silently disappear — intended here, catastrophic elsewhere | Read the third row twice. On a stream of account balances, a key going quiet for longer than the bound is an account nobody touched — and losing it is data loss that shows up as a missing row in a rebuilt view months later. On a stream of in-flight sessions, the same behaviour is the whole point: it is how the stream cleans up after keys that were abandoned rather than closed. ## Why this is an estate decision and not a per-stream tweak Three properties make it a decision to take deliberately and record: 1. **It is close to irreversible in one direction.** Enabling keyed retention lets the removal pass discard superseded values. Disabling it later does not bring them back; the history a downstream team assumed was there is gone, and nothing in the store remembers that it ever existed. 2. **It changes what the stream means to every reader.** A reader written on the assumption that it can see every value of a key breaks quietly under keyed retention — not by failing, but by computing a different answer. That is the worst kind of change to make unilaterally on a shared stream. 3. **It is constrained by the platform, not only by the design.** Designs genuinely differ: some platforms offer no keyed retention at all; some enforce bounds per subscriber rather than per stream, which makes a stream-wide keyed rule meaningless; a rented broker may fix a maximum you cannot raise, or expose neither choice. Establish what is available before designing around it. The practical discipline is small: write down, for each stream in the estate, which rule governs it and what promise that makes to readers — *complete history for N days*, *current value per key*, or *current value per key for keys still active within N days*. That last phrase is the one that has to be said out loud, because nobody infers it from seeing two rules enabled.