Messaging & Streaming

bootstrap.servers is the initial list of broker host:port pairs a client contacts to discover the full cluster. After the first metadata fetch the client learns all brokers, so the list only needs a few entries for redundancy.

A broker is a single Kafka server that stores topic partition data and serves produce/fetch requests. broker.id is its unique numeric identifier within the cluster; no two brokers may share the same id.

Each partition is an append-only log: records are only added to the end, never changed in place. Kafka splits that log into fixed-size files called segments so old data can be deleted or compacted one whole file at a time instead of editing one giant file.

The log cleaner is a background process that runs compaction: it scans a topic's log and keeps only the latest record per key, deleting older duplicates. You enable it by setting the topic config cleanup.policy=compact.

Kafka writes data to files and lets the OS keep recently used file pages in RAM (the page cache). It avoids a JVM heap cache to dodge GC pressure, double-buffering, and to reuse the OS cache that survives broker restarts.

Writers are pointed at the standby, readers restarted there, and only one site keeps accepting writes. The standby holds only what the copier had carried, and a reader's stored position from the source names a different record there.

A file backup fixes a stream at the instant it was taken, so every record written since is gone, along with every reader's progress. Streams keep moving while files do not, which is why the gap is counted in hours.

A cross-cluster copier is an ordinary client of both clusters: the source stores a record and answers the writer before the copier has even read it. The record therefore exists on the source first and on the target some time later.

The asynchronous copy hop sets the floor. Records acknowledged on the source cluster but not yet carried to the target exist in one place only, so losing the source loses them — the recovery point is at least the copy lag.

A write can be answered with no wait at all, once the leader holds it, once a majority of copies hold it, or once every caught-up copy holds it. Each rung up adds a network round trip of latency and removes one way to lose the record.