Architecture and Internals

When a segment rolls (closes), the leader broker's RemoteLogManager uploads it plus its indexes to remote storage and records metadata. Only after the upload is confirmed (COPY_SEGMENT_FINISHED) can the local copy be deleted, subject to local retention.

Setting broker.rack tags each broker with a failure-domain label (e.g. an AZ). Kafka then spreads a partition's replicas across distinct racks so one rack failure doesn't take all replicas down. It improves availability but doesn't change which replica is leader.

It sets how many threads, per log directory, Kafka uses to load and recover partition logs at startup and to flush them at shutdown. Raising it speeds up startup recovery on brokers with many partitions, especially after an unclean shutdown.

Under each path in log.dirs there's one directory per partition named <topic>-<partition> (e.g. orders-3). Inside are the segment files (.log/.index/.timeindex), a leader-epoch-checkpoint, and partition metadata. You inspect a .log with kafka-dump-log.sh.

The tail is the older, already-compacted part of the log (at most one record per key). The head is newer records appended since the last clean, which may still contain duplicates. The active segment — the one being written to — is always part of the head and is never compacted.

The cleaner makes two passes over the dirty head. First it builds an in-memory offset map: key -> highest offset seen. Then it recopies the log, keeping each record only if its offset equals the map's value for that key, and merges results into new compacted segments.

Each log segment has sparse index files (.index for offsets, .timeindex for timestamps) that Kafka memory-maps (mmap) so lookups happen in RAM. To find an offset, Kafka binary-searches the sparse index to get a nearby file position, then scans forward in the .log file.

Each delayed operation is filed in the purgatory under one or more watcher keys (typically TopicPartition). When an event affecting a key occurs — usually an ISR/high-watermark advance — the broker calls checkAndComplete on that key, which re-runs tryComplete on every operation watching it. Multiple keys let one request span several partitions.

queued.max.requests (default 500) is the capacity of the shared request queue between network threads and request-handler threads. When it's full, network threads stop reading new requests off sockets, which slows clients down — that is the backpressure.

Kafka first picks the right segment (the one whose base offset is the largest <= target). It binary-searches that segment's .index for the entry with the largest offset <= target, seeks to that byte position in the .log, then scans record batches forward until it reaches the requested offset.

The consumer sends a ListOffsets request with the target timestamp. The broker uses each segment's .timeindex (timestamp -> offset) to binary-search for the first offset whose timestamp is >= the target, then refines via the .index. It returns that offset and its timestamp.

flush.messages and flush.ms force the broker to fsync log data to disk after N messages or T milliseconds. By default both are effectively unset, so Kafka lets the OS flush dirty pages lazily and relies on replication, not local fsync, for durability — forcing frequent fsync hurts throughput.

A segment must be durably in the object store before its metadata is marked finished/readable; metadata must be the single source of truth for what's readable. Risks include orphaned objects, eventual-consistency reads, metadata/data divergence, and double-offload across leader changes.

A timing wheel adds and cancels a timeout in O(1), versus O(log n) for a heap/DelayQueue. Since brokers create and (usually) cancel a huge number of short-lived timeouts that complete early via events, O(1) insert/cancel matters far more than precise ordering.

The .txnindex is a per-segment file listing aborted transactions as ranges (producer id, first offset, last stable offset). With read_committed isolation, the broker sends this list so consumers can filter out records from aborted transactions and not read past the last stable offset.