What do flush.messages and flush.ms control, and why does Kafka discourage relying on broker fsync for durability?

## Dirty pages and fsync When the broker writes a record batch, the bytes go into the **page cache** as **dirty pages** — modified pages the kernel will eventually write to the physical disk during **writeback**. Until that happens, a power loss or kernel crash on that machine loses those bytes. The `fsync(2)` syscall forces all dirty pages of a file to the storage device and returns only when they are durable. fsync is **expensive**: it stalls until the device confirms, and it breaks the kernel's ability to batch many writes into efficient sequential flushes. ## The flush settings - **`log.flush.interval.messages`** (topic override `flush.messages`): fsync the log after this many messages have been appended. - **`log.flush.interval.ms`** (topic override `flush.ms`): fsync the log at most this often in time. - `log.flush.scheduler.interval.ms`: how often the flusher thread checks whether a flush is due. By **default these are set to effectively 'never'** (Long.MAX_VALUE), meaning Kafka does **not** proactively fsync on a schedule — it leaves flushing to the OS. ## Why Kafka prefers replication over fsync Kafka's durability model is **replication-based**: - A topic has `replication.factor` copies across brokers. - Producers set **`acks=all`** so the leader waits for all **in-sync replicas (ISR)** to receive the record before acknowledging. - **`min.insync.replicas`** sets how many replicas must be in sync for a write to be accepted; if fewer are available, the producer gets an error rather than a silent under-replicated write. Under this model, a committed record exists in the page cache of multiple brokers, ideally in **different failure domains** (racks/AZs). If one broker crashes and loses its un-fsynced pages, the data still lives on the surviving replicas, which become leader. So durability does **not** require each broker to fsync every write. This lets Kafka keep writes **sequential, batched, and OS-flushed** — the basis of its throughput. ## Why frequent fsync hurts - It serializes I/O and adds latency on the hot path. - It prevents the kernel from coalescing many dirty pages into large contiguous sequential flushes. - It can turn a bandwidth-bound workload into a latency/IOPS-bound one. Hence the docs explicitly recommend leaving flush intervals at defaults and relying on replication, tuning OS writeback (`vm.dirty_background_ratio`, `vm.dirty_ratio`, `vm.dirty_expire_centisecs`) if you want to smooth flush behavior. ## When you might tighten flush - **Single-broker / no replication** deployments where there is no replica to recover from. - **Correlated failures**: a whole-datacenter power loss can crash all replicas simultaneously, so 'another replica has it' fails — some operators fsync or place replicas across power/AZ domains to mitigate. - Regulatory/strict-durability requirements where even a rare un-fsynced loss window is unacceptable. ## Related durability levers (not flush) - `acks`, `min.insync.replicas`, `replication.factor`, `unclean.leader.election.enable=false` (don't elect an out-of-sync replica as leader and lose data), rack/AZ-aware replica placement. ## Edge cases / gotchas - fsync flushes data pages but Kafka also relies on consistent recovery on restart (log recovery, index rebuild) for partially written tails. - Setting `flush.messages=1` (fsync per message) is a classic anti-pattern that cripples throughput. - 'acks=all' guarantees replicas have it in **page cache/memory**, not necessarily fsynced — which is exactly the deliberate trade-off.