skip to content

A tier's count of deadline removals tripled this week while its forced-removal count stayed at zero - which number speaks to capacity?

level: middleimportance: must knowfreq 57%

answer

  1. one counter is about room
  2. sign matters, not magnitude
  3. deadline count follows writes and lifetimes
  4. zero forced is not proof of headroom

basics

~20 s

Only the forced-removal count speaks to capacity. A tripled deadline count tracks write volume and lifetime length, not room; a forced-removal count above zero means the store could not fit a write without taking an entry away.

solid answer

~50 s

The tripled number says nothing about room. Entries removed on their deadlines rise when more entries are written carrying lifetimes, when those lifetimes get shorter, or when a batch written together reaches its deadlines together - and adding memory would not move any of that. The other number is the capacity signal, and it signals by its **sign, not its size**: the store forces a removal only when it needed room and had none, so even a handful says the ceiling was reached and the store chose, on your behalf, what your service would stop finding. Zero over the week means the ceiling was never reached that week, not that headroom is comfortable. One caveat: check what each counter counts, because stores differ on how they book an entry already past its deadline when the room was needed.

go deeper

for a junior

Remember which number means what: entries removed on their deadlines is the application's own doing, entries the store had to remove is the tier saying it ran out of room.

for a middle

Explain the mechanics behind each number - write volume and lifetime length behind one, a write that would not fit behind the other - and why a larger tier moves only the second.

for a senior

Read the two together. A collapsing deadline count beside a non-zero forced count means the application stopped asking for removals and the store took over, which is the quiet version of this failure.

for a principal

Treat the definitions as part of the contract. Stores book a past-deadline entry taken under pressure differently and report at different granularity, so a fleet-wide capacity story cannot rest on one store's counter semantics.

## Two counters, not two views of one thing An in-memory tier that reports its removals usually reports them in two families: entries removed because a deadline passed, and entries removed because the store needed room. It is tempting to read them as one number split in two, as though both measured pressure at different intensities. They do not. They measure different things, and only one of them is about capacity at all. - **Removal on a deadline (expiry)** is the application's own instruction being carried out. It happens whether the tier is nearly empty or nearly full. - **Removal under pressure (eviction)** happens only when the store could not fit a write without giving something up. ## Why the deadline count cannot speak to capacity The count of entries removed on their deadlines moves for reasons that have nothing to do with how much room is left: - more entries are being written, and they carry lifetimes; - the lifetimes being attached got shorter, so the same traffic produces more removals; - a population of entries written close together reaches its deadlines close together, producing a burst; - a job that writes with lifetimes started running on a schedule. None of those is relieved by a larger tier. A tier at ten percent of its ceiling can produce an enormous deadline-removal rate quite happily - in fact a healthy rate here is usually a good sign, because it means entries are actually carrying lifetimes and something is taking them away. The mirror case is more interesting. If this count **falls to zero**, that is not reassurance. It usually means few or no entries carry a lifetime any more, and a keyspace of permanent entries has nothing that time will ever remove - which leaves the ceiling as the only thing that removes anything. ## Why any forced removal speaks to capacity The forced-removal count is unusual among operational numbers in that its **sign carries the information**. A store forces a removal only in one circumstance: it needed room it did not have. Two such removals in a day out of millions of writes is not noise, because the event itself is the finding - the ceiling was reached at least twice, and on each occasion the store, not you, decided which entry your service would stop finding. It is also a floor rather than a total. It counts the entries the store actually took; it does not describe how close the tier came on every other occasion, and a restart of the node resets it. ## Reading the pair together | Deadline removals | Forced removals | What the pair says | |---|---|---| | High | Zero | Entries carry lifetimes and time is removing them; nothing has been learned about room | | Low | Zero | Either little is being written or little carries a lifetime; the ceiling is untested | | High | Non-zero | Time is removing plenty and it still was not enough - what is live at once exceeds the tier | | Low | Non-zero | Almost nothing carries a lifetime, so the ceiling has become the only thing removing anything | The bottom row is the one that catches teams out, because the surface reading of the tier looks calm: few removals of either kind, a stable keyspace. What is actually happening is that the application stopped asking for removals and the store took the job over. ## Where the counting itself varies Stores of this class differ in ways that change what these numbers mean, so read the definitions before drawing the conclusion: - **Booking overlap.** An entry whose deadline has already passed, which the store finally takes when it needs the room, can be reported as a deadline removal on one store and as a forced removal on another. - **Granularity.** Some stores expose the two separately, some fold removals into one number, and some report per node rather than for the tier as a whole - in which case a busy node's finding can be diluted by quiet ones. - **Reset behaviour.** These are usually running totals since the process started, so a restart erases the history and a rate has to be computed over a window rather than read off directly. ## What each reading changes 1. **Deadline count moved, forced count still zero.** Nothing about capacity has changed. If the removals are hurting, the question is whether the lifetimes fit the access pattern - an application decision. 2. **Forced count left zero.** The tier has begun making room for itself. The follow-up question is what is live at once against how much the tier holds, and it is a different question from anything about lifetimes. 3. **Forced count non-zero and deadline count collapsing.** Look at what is being written without a lifetime, because permanent entries accumulate and the entries that do carry a lifetime end up paying for them.

  • The forced-removal count is two per day out of millions of writes. Is that ignorable?
    No. The event is the finding, not its rate: the ceiling was reached twice and the store picked what your service would lose. It is also a floor, since it says nothing about how close the tier came the rest of the time, and it resets when the process restarts.
  • The deadline-removal count has fallen to almost nothing. Is that good news?
    Usually not. It normally means few entries carry a lifetime any more, and permanent entries are removed by nothing at all. The tier then grows until the ceiling is reached, at which point the store's own behaviour - not the application's lifetimes - decides what goes.
  • Do the two counters always count disjoint events?
    No. An entry already past its deadline that the store takes because it needed the room sits between the two definitions, and stores book it differently. Check what each counter counts on the store in front of you before treating a movement in one as evidence about the other.

saying these in an interview costs you the question

  • A high deadline-removal rate means the tier needs more memory
  • Two forced removals a day is noise; wait for a spike
  • Both counters measure the same removal under different names
  • Zero forced removals proves the tier has comfortable headroom
  • Adding memory will bring the deadline-removal count down