Why does making a database commit genuinely durable cost latency, and what physically has to happen before the engine can acknowledge the commit?

## The layers a write passes through 1. **Database buffers** — in process memory. Lost on process crash. 2. **OS page cache** — a normal `write()` lands here. Survives the process dying, lost on kernel panic or power cut. 3. **Device write cache** — volatile DRAM inside the drive or controller, unless it is battery/capacitor backed. 4. **Non-volatile media** — platters or NAND. Survives power loss. Durability requires the transaction's record to reach layer 4 (or a layer-3 cache that is power-protected). Only the database can force that; a plain `write()` never does. The engine issues `fsync`/`fdatasync` (or opens with O_DSYNC, or uses the platform equivalent) and **blocks until the device answers**. That block is the latency. ## Why the wait cannot be optimized away for a single commit The device has to actually place the bytes somewhere that survives losing power, and report that it did. The magnitudes, roughly: - **HDD**: ~5–10 ms for a flush that requires rotation, sometimes a seek. - **Consumer SSD**: tens to hundreds of microseconds, highly variable, sometimes much worse under garbage collection. - **Enterprise SSD / NVMe with power-loss protection**: the on-device cache is capacitor-backed, so a flush can be acknowledged from cache — often tens of microseconds. - **Battery-backed RAID controller cache**: similar effect; the controller answers from protected DRAM. - **Cloud network-attached storage**: adds network latency to each flush; typically the dominant term. This is why 'durability is cheap on modern hardware' is only true when the hardware has power-loss protection. On a laptop SSD lying about flushes, durability is fast *and fake*. ## Per-commit, not per-byte The crucial mental model: durable commit cost is dominated by **flush count**, not data volume. Consequences: - 10,000 inserts in auto-commit mode = 10,000 flushes. The same inserts in one transaction = one. Batching is often a 10–100x throughput change with no other code difference. - Small transactions on a high-latency device are bound by the device's flush rate, so throughput per connection is roughly 1/flush-latency. More concurrent connections raise total throughput because independent commits' flushes can overlap and be serviced together. - Adding CPU or bandwidth does not help a flush-bound workload; changing the storage's flush characteristics does. ## Replication adds a second cost If 'durable' means 'a second machine has it', commit latency includes a round trip to that machine, plus that machine's own durability step if the configuration requires it. Same-rack is sub-millisecond; cross-availability-zone is single-digit milliseconds; cross-region is tens to over a hundred milliseconds — which is why cross-region synchronous commit is a deliberate, expensive design choice rather than a default. Some systems soften this by requiring only that the replica *received* the record into memory, not that it flushed it, trading a narrow failure window for much lower latency. ## What you can do about it - **Batch.** Fewer, larger transactions. The single biggest lever. - **Raise concurrency.** Independent transactions' flushes overlap, so total commit throughput scales beyond one connection's flush-latency limit. - **Buy power-loss-protected storage.** Turns a milliseconds-scale flush into microseconds legitimately. - **Segregate by value.** Keep strict durability for money and identity; consider relaxed modes for data whose loss window is genuinely acceptable. - **Never** solve it by disabling the device's flush honouring or mounting with barriers off. That converts a latency problem into silent data loss on power failure. ## Answering well Name the layers, say that only a device-honoured flush makes a commit durable, quantify roughly, and stress that the cost is per commit so batching and concurrency are the levers — then note that including a replica adds a network round trip to every commit.