Durability Options

## What "persistence is on" leaves out An in-memory store answers from memory, so everything it holds disappears when the process dies unless the store has been writing something to disk as it went. Saying persistence is "on" names a capability. The question anyone actually needs answered — a business owner, an incident review, an interviewer — is a different one: *when this process dies, how much of what we told callers had succeeded do we lose?* That is answered with one sentence carrying one number: **we may lose up to N seconds of acknowledged writes.** An **acknowledged write** is a write the caller was told had succeeded. The entire subject here is the gap between that acknowledgement and the moment the bytes are on disk — a write acknowledged to the caller but not yet flushed to disk is the exposure you are budgeting. ## The two things you must name to get a number 1. **The posture in force.** A store of this class is in one of four: keep nothing, keep a periodic whole copy, keep a replayed write log, or keep both. 2. **The interval that governs that posture.** For a periodic copy that is the copy interval; for a write log it is the flush policy — how often appended bytes are flushed to disk. Without both, "how much do we lose?" has no answer, because the honest answers range from *everything since the process started* to *the last fraction of a second*. | Posture | In memory after a restart | The window you can state | |---|---|---| | Keep nothing | an empty tier | every write since the process last started | | Keep a periodic whole copy | the keyspace as of the last copy | up to one copy interval of acknowledged writes | | Keep a replayed write log | the keyspace replayed from the log | up to one flush interval of acknowledged writes | | Keep both | whichever source the store restores from | that source's window — stores differ in which one they prefer and how they combine the two | ## Not every store has a posture to name Stores in this class differ here more than almost anywhere else, and the difference is not a setting. Some are built to keep nothing across a restart and offer nothing that changes that; on those the sentence is fixed — *a restart costs every write made since the process started* — and everything around the tier has to be designed for it. Others offer a periodic copy, a write log, or both, and some let a caller ask for stronger durability on an individual write. Asking "which posture?" and hearing "this store has none" is a complete answer, not a gap. ## The flush policy is the number that actually moves Where a write log exists, its flush policy has three granularities: - **flush on every write** — the smallest window, paid for on the write path by every single operation; - **flush on a timer (about a second)** — a window of roughly that timer; - **leave flushing to the operating system** — no bound you control, because the window is whatever the operating system had not yet written, which can be far larger than a second when the machine is under memory pressure. Notice the shape of that list: as the window shrinks, the cost moves onto the write path. That trade is what the sentence is really about. ## Writing the sentence 1. Name the posture and the interval that governs it. 2. State the window in seconds: *we may lose up to N seconds of acknowledged writes.* 3. Multiply by the write rate at the busiest moment, so seconds become records — "about three thousand orders" is something a non-engineer can judge; "one second" is not. 4. Say what those records are, and read the sentence to whoever owns them. ## The default nobody chose This is an interview question because most teams never made the choice. A store was installed, whatever it shipped with stayed, and the loss window is therefore an accident rather than a decision. That is what the interviewer is probing: not whether the window is one second or five minutes, but that nobody can say which, and nobody has put the sentence in front of the person who would have to live with the consequence. ## What the sentence does not promise Persistence here bounds a restart. It decides whether the tier comes back empty or nearly current, and how much replay at start costs in recovery time. It does not turn the tier into a durable system of record: under most flush policies the caller is told the write succeeded before the bytes are safe, the disk beneath it is a single failure domain, and the whole component is built around memory as its medium. The sentence is a budget for a bounded, survivable loss — which is exactly why this tier is never the only copy of anything that matters. Keep the scope honest, too. The sentence describes a crash or restart of this single process. An acknowledged write lost because another node was promoted without it has a different cause and a different remedy, and does not belong in this number.