How does a Kafka broker know on startup whether it was shut down cleanly, and why does that distinction matter?

## The problem When a broker restarts, it needs to know whether the data on disk is in a known-good, fully-flushed state, or whether it might be partially written because the process died mid-flight. Recovery (re-scanning and validating the tail of each log) is correct but slow, so Kafka wants to skip it whenever it safely can. ## The clean-shutdown marker Kafka solves this with a **marker file written per log directory** during an orderly shutdown, named **`.kafka_cleanshutdown`** (a hidden file in each `log.dirs` entry). The shutdown sequence is roughly: 1. Stop accepting new requests. 2. Flush all partition logs to disk (fsync). 3. Advance the `recovery-point-offset-checkpoint` (and write the other checkpoints) so the recovery point equals the log end offset. 4. Write the `.kafka_cleanshutdown` marker into each directory. ## On startup For each log directory the broker checks for the marker: - **Marker present →** the previous shutdown was clean. The broker deletes the marker (so a later crash without re-writing it will be detected) and loads logs using the trusted checkpoints. No tail re-validation is needed → **fast startup**. - **Marker absent →** the previous stop was unclean (kill -9, OOM, crash, power loss). The broker performs **log recovery**: it re-scans each affected partition from its recovery point to the end, validating record batches and rebuilding/ truncating indexes as needed. This is the slow path, parallelized by `num.recovery.threads.per.data.dir`. ## Why the distinction matters - **Startup time / availability:** clean shutdown → seconds; unclean shutdown on a many-partition broker → potentially many minutes of recovery, during which those partitions are unavailable on this broker and under-replicated. - **Operational practice:** always stop brokers with a graceful shutdown (SIGTERM and let it drain) rather than SIGKILL, and set `controlled.shutdown.enable=true` so leadership migrates off the broker before it stops — this both avoids recovery and avoids client-visible disruption. ## Edge cases - The marker is **per directory**, so in a JBOD setup one disk could be clean while another (that failed) is treated as unclean. - If a shutdown is interrupted *after* flushing but *before* writing the marker, the broker conservatively runs recovery — slower but safe. - Recovery is bounded by the recovery point, so even an unclean restart only re-scans the unflushed tail, not the entire log. ## Bottom line The `.kafka_cleanshutdown` marker, written per log dir at orderly shutdown and deleted at startup, is the signal that lets a broker skip recovery. Its presence means 'trust the checkpoints, start fast'; its absence means 'I might have crashed — recover the log tails.'