Architecture and Internals

A broker is a single Kafka server that stores topic partition data and serves produce/fetch requests. broker.id is its unique numeric identifier within the cluster; no two brokers may share the same id.

Each partition is an append-only log: records are only added to the end, never changed in place. Kafka splits that log into fixed-size files called segments so old data can be deleted or compacted one whole file at a time instead of editing one giant file.

The log cleaner is a background process that runs compaction: it scans a topic's log and keeps only the latest record per key, deleting older duplicates. You enable it by setting the topic config cleanup.policy=compact.

Kafka writes data to files and lets the OS keep recently used file pages in RAM (the page cache). It avoids a JVM heap cache to dodge GC pressure, double-buffering, and to reuse the OS cache that survives broker restarts.

Each Kafka log segment has a .index file mapping a message offset to its physical byte position in the .log file, and a .timeindex file mapping a timestamp to an offset. They let Kafka jump near a record without scanning the whole segment.

Tiered Storage (KIP-405) moves old, closed log segments from local broker disks to cheaper remote object storage (like S3). Recent data stays local for fast reads; older data is fetched from remote when needed.

listeners is where the broker binds and accepts connections. advertised.listeners is the address it publishes in metadata for clients to connect back. listener.security.protocol.map maps each named listener to a security protocol (PLAINTEXT, SSL, SASL_SSL, etc.).

Give the Kafka broker a small JVM heap (commonly 6 GB) and leave most of the machine's RAM free for the OS page cache, which caches log file data. Kafka relies on the page cache, not heap, for read/write throughput.

A tombstone is a record with a real key but a null value, signaling 'this key is deleted'. The cleaner keeps tombstones around for delete.retention.ms (default 24h) so consumers can observe the deletion, then removes them in a later pass.

If a fetch can already return at least fetch.min.bytes of data, the broker replies immediately. If not, it parks a DelayedFetch and waits up to fetch.max.wait.ms for more data to accumulate; if that timer fires first, it returns whatever it has (possibly empty).

The leader writes the records to its log, then parks a DelayedProduce in purgatory keyed by each target partition. When followers replicate and the high watermark advances past the produced offsets for all partitions, the operation completes and the broker acknowledges the producer.

An acceptor thread accepts the TCP connection and hands it to a network (processor) thread, which reads the request and puts it on a shared request queue. A request-handler (I/O) thread takes it, KafkaApis processes it, and the response goes back through the network thread to the client.

RemoteStorageManager moves the actual segment bytes to/from remote storage (the data plane). RemoteLogMetadataManager tracks metadata — which segments exist remotely, their offset/epoch ranges, and their state (the control plane).

The controller is the broker/node responsible for cluster-wide management: partition leader election, tracking broker liveness, and propagating metadata. In ZooKeeper mode one elected broker is the controller; in KRaft, dedicated controller nodes run a Raft quorum that stores metadata in a log.

Each log directory holds small checkpoint files. recovery-point-offset-checkpoint records, per partition, the offset up to which data has been flushed to disk (so recovery can skip it). replication-offset-checkpoint records the high watermark, the offset up to which messages are replicated/committed.

A segment rolls when it hits segment.bytes (default 1 GiB) or segment.ms (default 7 days) — whichever comes first. Smaller values roll more often, giving finer retention but more files and open handles; larger values mean fewer, bigger files.

Each cleaner thread picks the log with the highest 'dirty ratio' — the fraction of the cleanable log that hasn't been compacted yet. A log only becomes eligible once that ratio reaches min.cleanable.dirty.ratio (default 0.5).

Zero-copy lets Kafka send file data from the page cache straight to the network socket using the sendfile syscall (via Java's FileChannel.transferTo). It skips copying the bytes into the application (user space) and back, saving CPU and memory bandwidth.

num.network.threads sizes the network (processor) pool that does socket I/O; num.io.threads sizes the request-handler pool that does the actual work (log reads/writes). Use the idle-percent metrics: if a pool's average idle percent is low, it is saturated and you raise its thread count.

The broker sees the requested offset is below the local log's start, asks the RemoteLogMetadataManager which remote segment holds it, uses the remote offset index to find the byte position, streams the bytes back via the RemoteStorageManager, and returns them in the fetch response — transparently to the consumer.

On a clean shutdown the broker flushes all data, updates the checkpoint files, and writes a clean-shutdown marker file in each log dir. On startup, if that marker is present it skips log recovery; if it is missing (a crash), it runs recovery.

Purgatory is a broker-side holding area for requests that can't be answered immediately. Instead of blocking a thread, the broker parks the request there and completes it later when a condition is met or a timeout expires.

The acceptor thread only accepts new client connections (one per listener). Network/processor threads handle reading and writing data on existing connections; there are num.network.threads of them (default 3), and the acceptor spreads new connections across them round-robin.

Setting multiple paths in log.dirs (JBOD = Just a Bunch Of Disks) lets one broker spread partition data across several independent disks for more total capacity and I/O. Kafka assigns each new partition to the directory with the fewest partitions.

A segment's files are named after its base offset — the offset of the first record in that segment — zero-padded to 20 digits, e.g. 00000000000000006000.log. So the filename tells you the lowest offset that segment can contain.

Within a single partition, offsets are strictly increasing and assigned in append order — record N+1 always has a higher offset than record N. Across partitions there is no ordering. Segments preserve this because each new segment's base offset continues where the previous one ended.

Kafka only appends to the end of partition log files instead of writing in random places. Sequential writes are far faster than random writes (no disk seeks), so even HDDs can sustain high throughput, and writes go through the page cache first.

KafkaApis is the broker class that actually handles requests. A request-handler thread calls KafkaApis.handle(), which looks at the request's API key (Produce, Fetch, Metadata, etc.) and dispatches to the matching handler method.

index.interval.bytes (default 4096) is how many bytes of log Kafka appends before adding a new index entry. Smaller means denser indexes and faster lookups but larger files; larger means sparser indexes, smaller files, slightly slower lookups.