A broker acknowledges a record as stored, yet a power cut on that machine loses it — why?

## Two events that share one word When a writer sends a record and the broker answers "stored", two quite different things may or may not have happened to those bytes. - **Acceptance.** The broker hands the bytes to the operating system with an ordinary write. The operating system copies them into its **file cache** — ordinary volatile memory that it manages — and returns at once. Nothing has reached a physical device yet. The broker is now free to answer the writer. - **Forcing.** A separate, explicit instruction tells the operating system to push those cached bytes onto persistent media and not to return until the device reports them stored. Nothing about an acknowledgement implies the second event happened. On most brokers and streaming platforms in this class, the answer to the writer is sent after acceptance, and forcing runs on its own schedule behind it. ## Committed is not durable It is worth keeping two words apart deliberately, because the whole subject lives in the gap between them. - **Committed** means the acknowledgement rule was satisfied — whatever the operator asked the server to wait for before answering: no wait at all, the leader alone, a majority of copies, or every caught-up copy. - **Durable** means the bytes are on persistent media and will still be there after the machine loses power. A record can be fully committed under the strictest acknowledgement rule available and still be durable nowhere. | State of a record | Survives the broker process dying | Survives the machine losing power | |---|---|---| | Accepted into the file cache on one node | Yes — the operating system still holds it | No | | Accepted into the file cache on several nodes within one cluster | Yes | Only if those machines lose power separately | | Forced onto persistent media | Yes | Yes | The middle row is the one that surprises people. Extra copies genuinely protect the record, because a machine that crashes on its own leaves the others holding the bytes and able to write them out. They protect it much less when the machines go down together, because each of them was holding an unforced span in memory at that moment. ## Sizing the loss window The **loss window** is the span of records that were accepted but not yet forced. It has two useful units, and an operator should be able to quote both: 1. **Time** — how long it has been since the last force. If the platform forces roughly once a second, the window is roughly a second of ingest. 2. **Bytes or records** — that time multiplied by the write rate on that node. A node taking 50 MB per second with a one-second forcing policy is carrying up to about 50 MB of exposed data. Both numbers move with load: the same forcing policy produces a far larger window at peak than at night. That is why a durability posture written in seconds is easier to reason about than one written in records. ## What varies between platforms This is a place where platforms in the same class genuinely differ, and an interview answer that ignores that reads as one product's manual: - Some platforms expose a forcing policy as a per-stream override on top of a cluster default; others expose only a cluster-wide one; a rented cluster may expose none at all and simply publish what it does. - On designs where durability comes from an underlying shared replicated store rather than from per-node copies, the broker is not the thing making this decision. The store's own durability contract is what the acknowledgement inherits, and the question becomes what that contract says. - Broker designs that keep exactly one mirrored copy answer once the pair holds the record — which is still memory on both machines unless something forced it. - Hardware changes the price without changing the meaning. A device whose own cache is protected against power loss can confirm a force very quickly, so forcing costs less there; it does not make an unforced write safe. ## What an operator does with this 1. Find out what the platform actually does before it answers a writer, and whether that is configurable. 2. State the loss window in seconds and in bytes at peak ingest, per node. 3. Decide whether an event that takes several of those machines at once is in the threat model for this stream. 4. Record the answer as part of the stream's durability posture, next to its copy count and its acknowledgement rule, rather than leaving it as an accident of the defaults.