skip to content

What do the flush.messages and flush.ms settings control, and why are they usually left at their defaults?

level: seniorimportance: should knowfreq 45%

answer

  1. flush.messages = fsync every N msgs
  2. flush.ms = fsync every T ms
  3. defaults ≈ infinite → OS decides
  4. replication, not flush, gives durability
  5. lowering them = throughput hit, smaller loss window

basics

~20 s

flush.messages and flush.ms tell a broker to fsync its log to disk after a certain number of messages or after a time interval. They're usually left at defaults because Kafka relies on replication for durability, so forcing extra fsyncs just hurts throughput.

solid answer

~40 s

These are broker/topic-level knobs that govern how aggressively Kafka forces its log to physical disk. `log.flush.interval.messages` (topic override: `flush.messages`) triggers an fsync after that many messages are appended to a partition; `log.flush.interval.ms` (topic override: `flush.ms`) triggers an fsync after that many milliseconds. Their effective defaults are essentially infinite/unset, meaning Kafka does NOT force flushes and instead lets the operating system's background dirty-page writeback handle persistence. They stay at defaults because durability comes from replication (acks=all + min.insync.replicas across independent brokers), not from disk flushes. Lowering them shrinks the per-broker data-loss window on power failure but adds synchronous fsync cost that reduces throughput and increases latency. In well-replicated clusters that trade-off isn't worth it; the OS flush plus replication is sufficient.

go deeper

for a junior

Know these settings force the broker to write its log to disk after N messages or T milliseconds, and are usually left alone.

for a middle

Explain that defaults are effectively off and the OS handles flushing because durability comes from replication.

for a senior

Reason about the throughput-vs-loss-window trade-off and when (low RF, edge) tuning them is justified.

for a principal

Decide cluster-wide flush policy in concert with replication factor, OS writeback tunables, and storage characteristics; teach the replication-first model.

## What these settings are Kafka stores each partition as an append-only log on the broker's filesystem. When records are appended, the bytes go into the **OS page cache** (RAM) and are written to physical disk later by the kernel. `flush.messages` and `flush.ms` let you force that disk write (`fsync`) sooner: - **`log.flush.interval.messages`** (broker default) / **`flush.messages`** (per-topic override): force an `fsync` on a partition's log after this many messages have accumulated since the last flush. - **`log.flush.interval.ms`** (broker default) / **`flush.ms`** (per-topic override): force an `fsync` after this many milliseconds have elapsed. There's also `log.flush.scheduler.interval.ms`, the period at which Kafka's background thread checks whether the message/time thresholds have been hit. ## What the defaults are The message-count default is effectively `Long.MAX_VALUE` (never trigger on count) and the time default is unset (`null` → never trigger on time). Net effect: **Kafka does not proactively fsync; it leaves flushing to the OS.** The kernel writes dirty pages to disk on its own schedule (tunable via OS settings like `vm.dirty_ratio`, `vm.dirty_expire_centisecs`). ## Why leave them alone? Kafka's durability model is **replication-first**. With `acks=all`, `replication.factor=3`, and `min.insync.replicas=2`, an acknowledged record exists in the page cache of multiple independent brokers. The probability that *all* of them lose their copy before any flushes is low, because they're typically on separate hosts/racks/AZs with independent power. Against that backdrop, forcing frequent fsyncs: - **Costs throughput and latency** — each `fsync` is a synchronous round-trip to the storage device, and aggressive flushing serializes I/O. - **Buys little extra safety** — it only shrinks the per-broker page-cache loss window, which replication already covers for non-correlated failures. ## When you *might* tune them - Very low replication factor (e.g., RF=1) where you have no replica safety net. - Single-broker or edge deployments. - A regulatory requirement that data be on stable storage before considering it durable, despite the throughput cost. Even then, the modern guidance is usually to fix the replication setup rather than to fsync per message. ## Key nuance Flushing controls *when bytes reach the platter on a given broker*; it does **not** change *when the producer is acknowledged*. The producer ack is governed by `acks` and ISR replication, which are independent of `flush.messages`/`flush.ms`.

  • Does setting flush.ms low change when the producer gets its acknowledgement?
    No. The producer ack is determined by acks and ISR replication. flush.ms only changes when a broker forces its own log to disk; the two are independent.
  • What OS-level controls also affect when dirty pages reach disk?
    Kernel writeback tunables like vm.dirty_ratio, vm.dirty_background_ratio, and vm.dirty_expire_centisecs govern when the OS flushes dirty page-cache pages, which is what handles flushing when Kafka leaves flush.messages/flush.ms at defaults.

saying these in an interview costs you the question

  • Thinking flush.messages/flush.ms control producer acknowledgement timing — they don't.
  • Recommending aggressive per-message flushing as a best practice for a properly replicated cluster.
  • Believing the defaults force a flush — they effectively disable forced flushing.

context

resolving links…