A store's memory ceiling is set to the container's full 16 GB limit — why is that fatal, and what belongs in the gap?

## Two limits, enforced by two different parties An in-memory store that offers a **memory ceiling** is offering you a limit *it* enforces, against *its own* accounting of what it is holding. The **machine limit** — the container's memory limit, or the physical memory of the host — is enforced by the operating system, against everything the process holds. That difference is the whole subject. The store's ceiling is a number it can act on politely and in advance. The machine limit is a number enforced by failure: the process slows to a crawl as memory is paged to disk, and then the operating system's out-of-memory kill ends it. Setting the first equal to the second means the store can never act first. ## What lives in the gap The process holds memory on behalf of things that are not entries. Three consumers are worth naming because they are the ones that surprise teams, and because each one is large, bursty, and driven by something other than how much data you stored: 1. **Replication buffers.** Where the tier has followers, the primary holds writes on their behalf — enough to feed a follower that is behind, and enough to carry one through a re-connection. The size is driven by write rate and by how far behind a follower is allowed to fall, not by the size of the dataset. 2. **Client output buffers.** A server can produce a reply faster than a connection drains it. The undelivered bytes queue per connection, so a slow reader, a stalled network path, or a caller that asked for a very large reply parks memory on the server. This grows with the number of connections and with the size of the largest reply, and it is a common reason a process that had "plenty of headroom" died. 3. **Transient duplication during a whole-keyspace background copy.** On stores that produce a point-in-time copy by writing out the whole keyspace while still serving traffic, pages modified during the copy are duplicated, so the process can hold meaningfully more than the data size for the duration. The extra is proportional to how much is written while the copy runs, not to the dataset — a quiet tier duplicates almost nothing and a write-heavy one can duplicate a large fraction. | Consumer | Held on behalf of | Applies when | What makes it grow | |---|---|---|---| | Replication buffers | Followers | The tier has replicas | Write rate; how far a follower may lag | | Client output buffers | Connections | Always, but sharply with slow readers | Connection count; largest reply size | | Copy-time duplication | The background copy | The store writes whole-keyspace copies | Write rate while the copy is running | These three are not an exhaustive list. What the allocator holds beyond the store's own accounting also sits in the gap, and so does anything else sharing the box. That is an argument for the gap being larger, never smaller. ## Why the ceiling at the machine limit inverts the failure With a gap, growth in stored data reaches the store's ceiling first, and the store gets to apply whatever posture it was configured with — the three possible outcomes at the ceiling are a subject of their own, and the point here is only that the store *has* an outcome to apply, in its own process, on its own terms, with its own logging. With no gap, none of that happens. The buffers and the copy push the process past the machine limit while the store's accounting still reads "under the ceiling", and the failure is delivered by the operating system: first the slowdown as memory is paged to disk — which looks to callers like the tier suddenly became a slow disk — and then the kill, which takes everything the tier held with it. The sentence to be able to say out loud: *a ceiling set at the machine limit converts a memory problem the store could have handled into a process death it cannot.* ## Where the machine limit is not the only shape - In a container with a hard memory limit, exceeding it kills the process, usually with no paging first — the slowdown warning you expect may simply not arrive. - On a shared host with paging configured, the process degrades for a long time before anything is killed, and the tier's latency is unrecognisable long before that. - Either way, the number worth alerting on is the process's resident size against the machine limit, *in addition to* the store's own accounting against its ceiling, precisely because the two can diverge. ## What varies between stores - Not every store offers a ceiling at all. Where it does not, the headroom argument moves into the node choice: you pick a machine whose limit leaves the same gap above what you intend to hold, and you watch it rather than configure it. - Stores differ in exactly which memory falls inside the ceiling's line — some count more of their own bookkeeping and buffers than others — so a gap sized by "what I know the setting excludes" is fragile, while a gap sized by named consumers and then verified against the process's resident size is not. - Not every deployment has all three consumers. No followers means no replication buffers; a store that never writes a whole-keyspace copy has no copy-time duplication. Headroom is defended item by item, not as a blanket percentage.