Cluster Linking and Mirror Topics

## The problem Kafka clusters often need a copy of their data in another cluster — for disaster recovery (DR), migrating to a new cluster, or serving readers in another region. Historically this was done with **MirrorMaker 2 (MM2)**, which runs on **Kafka Connect**: a separate cluster of worker processes that consume from the source and produce to the destination. That works, but it has moving parts (a Connect cluster, connector tasks, consumer groups) and — importantly — it does **not** preserve offsets, because re-producing assigns brand-new offsets on the destination. ## What Cluster Linking is **Cluster Linking** moves replication *into the broker itself*. Instead of an external process, the **destination cluster's brokers connect directly to the source cluster's brokers** and fetch data, behaving much like a follower replica fetches from a leader. The named connection between the two clusters is called a **cluster link**. Key concepts: - **Cluster link**: a configured, named connection object created on the **destination** cluster that points at the source (bootstrap servers, security config). One link can carry many topics. - **Mirror topic**: a topic on the destination that is bound to a source topic via the link. It is **read-only** — producers cannot write to it — and it continuously pulls new records from the source. - **Byte-for-byte / offset-preserving**: record at offset N on the source becomes the record at offset N on the destination. Partition count, ordering, and offsets all match. ## Why offset preservation matters Because offsets match exactly, a consumer that was at offset 5000 on the source can resume at offset 5000 on the destination and read the *same* message. This is what makes Cluster Linking suited for DR failover and live migration — consumer position is meaningful across clusters. MM2 cannot do this natively (it needs offset-translation via a checkpoint topic, which is approximate). ## How it differs from Connect-based replication | Aspect | Cluster Linking | MM2 / Connect replicator | |---|---|---| | Runtime | Built into the broker | Separate Connect cluster | | Offsets | Preserved exactly | Re-assigned (translation needed) | | Mirror topic | Read-only until promoted | Writable normal topic | | Operational overhead | Low (no extra cluster) | Higher (workers, tasks) | ## Edge cases / caveats - A mirror topic **cannot be written to** while it's mirroring; you must **promote** (or fail over) it to make it writable. - The source topic's config (partitions, etc.) is mirrored; you don't manage partitions independently on the destination. - Cluster Linking is a **Confluent** feature (Confluent Platform / Confluent Cloud), not part of vanilla Apache Kafka.