One store holds recomputable page fragments alongside leases and deduplication records under a single ceiling and removal policy. What is wrong, and what would you do?

## The mismatch A removal policy is a property of the store, or at best of an entry's eligibility marking. Criticality is a property of the data. When one keyspace holds several populations, one policy is applied to all of them, and the policy that is correct for the cheapest population is applied to the most expensive one. The same mechanism produces three different failures: | entry removed | what it costs | who notices | |---|---|---| | a recomputable fragment | one extra fetch and some latency | nobody, until the origin is hot | | a lease held by a worker | two workers act as though each holds it | whatever the lease protects | | a deduplication record | a retried request is processed twice | whoever receives the duplicate effect | No tuning of the order axis addresses this. A better ranking removes a better fragment and a better lease. ## Why it is usually invisible until it is not A mixed tier behaves perfectly while it has headroom, because removal never fires. The populations also grow at unrelated rates: the fragments grow with traffic, the leases with concurrency, the deduplication records with retention. So the first time the tier reaches its ceiling is typically long after the design decision, under load, and the symptom is not a memory alert but a correctness incident somewhere else entirely - a duplicated side effect, or two workers in the same critical section. ## The remedies, strongest first 1. **Separate the populations into separate stores.** Distinct ceilings, distinct postures, distinct failure domains. This is the only arrangement in which the fragment workload cannot consume the memory the leases need, and it is worth real operational cost. Its price is another thing to run, another connection pool, and another tier to watch. 2. **Use the eligibility axis, not the order axis.** If one tier is unavoidable, attach lifetimes to the disposable population, leave the irreplaceable entries without one, and configure the store for lifetime-only removal. Removal then lands entirely on entries the application marked as expendable. This depends on the store supporting the restriction at all, and it has its own failure: when nothing is eligible, the ceiling shows up as writes that cannot be served. 3. **Bound the irreplaceable population by design.** The entries you refuse to lose must not be allowed to grow without limit, or remedy 2 converts one failure into another. Cap what is retained and keep the permanent share small enough that the expendable pool can always absorb pressure. 4. **Choose the failure you prefer, explicitly.** Failing writes while reads continue is the honest posture when correctness dominates: it is loud, it is visible to callers, and it degrades service rather than data. The reverse argument is legitimate too - a tier whose job is to absorb load for a fragile origin may be better off removing something than refusing, and a lost lease that can be re-acquired may genuinely be cheaper than a stalled request path. What is not legitimate is having never made the choice. ## What a principal-level answer adds - **A measurement, not an estimate.** Attribute the resident memory to each population rather than reasoning about it, and know the share of the ceiling each one holds before choosing a posture. - **An alert that fires before the ceiling, not at it.** The useful signals are removals per minute and the eligible share, not just bytes: on a tier holding irreplaceable state, a removal rate *is* a data-loss rate, and it should page someone whether or not memory has moved. - **A named blast radius per population.** Being able to say what one removal costs for each kind of entry is what converts this from a configuration discussion into a design one. ## Do not assume the mechanism you are used to The options above depend on capabilities that vary across this class of store: - Whether there is a choice of removal family at all, or one fixed behaviour. - Whether removal can be restricted to entries carrying a lifetime. - Whether the store has a ceiling concept, or simply grows until the machine intervenes - in which case the operating system chooses what dies, and it chooses the whole process. - Whether removal ranks over the whole keyspace or within the fixed-size block class that needs room. - What the **default** is, which is the setting most tiers are actually running. So the design step is to verify the posture of the store in front of you, state it, and only then argue the arrangement. Presenting one store's defaults as the model is how a design review approves an arrangement that the deployed store does not implement. ## The short version One ceiling plus one policy means one blast radius for entries with very different values. Split the tier if you can; if you cannot, make the irreplaceable entries ineligible, keep them small, watch the eligible share, and write down which failure you chose at the ceiling.