A tenant on a shared in-memory store deletes its entire prefix at peak and every other application times out - why, and how should it have been done?

## Why a tenant's own cleanup is everyone's incident The tenant did nothing conceptually wrong: those were its entries, and removing them is the correct end of their life. What makes this an incident is that **the work is done by a server nobody divided up**. A shared instance has one pool of execution capacity, and a bulk removal is a large amount of work submitted into it in a short time. The neighbours did not send that work, cannot see it, and have no queue of their own to be served from. Three separate costs are in play, and a good answer separates them: 1. **Finding the entries.** Unless the tenant already knows every key, the keys have to be located. A single operation that returns the whole keyspace builds one reply proportional to the number of entries, holds the server while it does so, and then ships it - on a busy instance that alone is the outage. 2. **Removing them.** Each removal is cheap; a few million of them are not. The instance is now serving one tenant's cleanup and everyone else's traffic from the same capacity. 3. **Giving the memory back.** Freed entries return memory to the process's allocator, and the resident footprint the operating system sees does not necessarily fall with the stored-data size. A co-tenant watching the wrong number concludes nothing happened. ## Where the variation is The first instinct - 'the server runs one operation at a time, so everything blocks' - is true of one design, not of the class: - On **single-execution-thread** servers, one long operation is a stop-the-world event for every caller of that instance, however unrelated their keys. - On **multi-threaded** servers that reach per-operation atomicity by locking the entry, unrelated callers can make progress, but the cleanup still consumes cores, memory bandwidth and connection capacity. Bulk work saturates a multi-threaded server perfectly well; it just degrades more gradually. - If the instance has a **copy** following it, every removal is propagated. A copy that was keeping up now has a long stream to apply and falls behind, and callers reading from it see entries the primary no longer holds. On stores that acknowledge a write only after a copy holds it, the primary's own writes slow instead of the copy drifting. ## The safe procedure on a live shared tier 1. **Agree the window.** On a shared instance this is a change to a system other teams depend on. Tell the co-tenants what will run, when, and what to watch. The point is not politeness - it is that they, not you, will see the symptom first. 2. **Locate incrementally.** Walk the keyspace with a cursor in bounded batches rather than asking for all of it at once, so the server's work per call stays small and predictable. 3. **Remove in batches with pauses.** A batch size and a pause between batches are the two knobs that turn an outage into a background task. Start small, watch, then increase. 4. **Measure from the caller.** Watch latency as the co-tenants experience it, including time spent waiting for a connection. The server's record of operations that exceeded a time threshold records execution only, so it can look untroubled while callers are timing out. 5. **Stop on a threshold, not on a feeling.** Decide beforehand what caller-observed latency ends the run, and be willing to finish tomorrow. ## What each approach costs | Approach | Effect on co-tenants | When it is defensible | |---|---|---| | One operation that returns the whole keyspace, then remove everything | Worst: a long hold plus an enormous reply, at peak | Never on a live shared instance | | Remove the whole prefix in one pattern-matched operation | Long single unit of work the neighbours queue behind | Small keyspaces, off peak, on an instance you own alone | | Incremental traversal, batched removals, pauses | Bounded and tunable; can be stopped at any point | The default on any shared tier | | Restart or empty the instance | Total: every tenant loses everything at once | Only with every occupant's agreement | ## The platform's part of the failure It is worth saying out loud in an interview that the tenant is not the only party at fault. **This class of store generally offers no per-tenant rate limit**, so nothing prevented an ordinary cleanup from becoming a shared outage. What a platform can do is route tenants through a client library or proxy it controls and gate large removals there, publish a cleanup procedure with batch sizes and a window, and move any tenant whose cleanups are routine onto its own instance - the one bound that does not depend on everyone remembering the procedure. ## Afterwards Record the caller-observed latency during the run and the batch size that produced it, because the next cleanup is sized from that number rather than from a guess. And check the resident footprint against the stored-data size before anyone concludes the memory was not actually released.