A store set to remove only entries carrying a lifetime reaches its memory ceiling while every entry is permanent: what happens to the next write?

This is the dead end in the middle of the three outcomes: a store that was configured for one of them delivers another, and the configuration still reads exactly as the operator intended. ## The eligibility rule Removal under memory pressure can be restricted by **which entries are eligible**. The two positions are: - **Whole-keyspace removal** — any entry may be taken, including entries that will never expire on their own. - **Lifetime-only removal** — only entries that carry a lifetime may be taken; entries with no lifetime are off limits. The second exists for a good reason. An entry given a lifetime was, by definition, written by someone who accepted that it would go away. An entry with no lifetime is usually the opposite: something meant to stay, often the only copy of the fact it records. Restricting removal to the first group is a deliberate promise that pressure will not destroy the second. ## What happens when nothing is eligible The promise is kept, and the consequence is the dead end. The store reaches its memory ceiling, looks for a candidate, and finds none, because every entry it holds is permanent. It does not widen the search — widening it would break the exact promise the setting was chosen to make. With no way to free memory, the only thing left is to **refuse the write**. From the caller's seat this is indistinguishable from a store configured to refuse from the start: - writes fail, and the caller is told; - reads of existing entries generally keep succeeding; - nothing already stored is lost. The store is, in effect, running a posture nobody selected. ## Why this is confusing in production The confusion is that every layer reports something true and misleading. | What you observe | What it seems to mean | What it actually means | |---|---|---| | Removal is configured | Pressure will be absorbed | Pressure will be absorbed *if* something is eligible | | Removal count is flat | There is no memory pressure | There is pressure and no candidate | | Data size sits at the ceiling | The ceiling is working | The ceiling is being enforced by refusal | | Writes are failing | Something changed in the write path | Nothing changed except which entries exist | The path in is usually gradual and invisible. A keyspace starts out mostly lifetimed, and over months a workload is added whose entries are written permanently — a registry, a set of flags, something a batch job populates once. Each such write reduces the eligible population slightly. The store keeps absorbing pressure normally until the day the last eligible entry is taken, and then it stops absorbing anything. ## The diagnosis The combination that identifies it, and distinguishes it from a store that was simply configured to refuse: 1. Writes are failing and reads are fine — so this is the refusal outcome, whatever the configuration says. 2. The memory the store attributes to its entries is pinned at the ceiling — so this is pressure, not a caller problem. 3. The removal count is flat rather than rising — so removal is configured but not happening. 4. The eligible population is empty or near it — the proportion of entries carrying a lifetime has collapsed. Points 3 and 4 are what separate this from an ordinary refusal posture. Note that the trigger throughout is the **ceiling**: this is eviction that could not find a victim. Removal because a deadline passed is expiry, a different event, and it does not depend on memory pressure at all. ## The two repairs, and what each costs - **Give entries a lifetime.** The eligible population returns and the store absorbs pressure again — but you have re-entered the removal outcome deliberately, and anything you have just made removable can now vanish silently under pressure. That is a correct choice only for entries with a source of truth. - **Accept refusal and treat it as the posture.** Alert on write failures, and keep the permanent entries safe by design. What the application should then do with a refused write is its own subject. Raising the ceiling is not a third repair; it postpones the same day. ## What varies Do not assume this shape exists everywhere. Whether a store distinguishes eligible from ineligible entries at all varies across the class: some will remove any entry under pressure and offer no way to protect one, and some do not remove at all, in which case refusal is the only posture and there is no dead end to fall into.