skip to content

You're designing active/passive disaster recovery across two regions with Cluster Linking. Walk through the topology, failover, fail-back, and the key constraints/trade-offs you must account for.

level: principalimportance: should knowfreq 30%

answer

  1. A active, B passive read-only mirror + offset/ACL sync
  2. Promote (RPO 0) vs failover (tail loss); no auto-failover
  3. Fail-back = NEW reverse link, B has diverged
  4. Links unidirectional → no naive active-active
  5. Replicate Schema Registry separately; watch EOS

basics

~20 s

Run primary in region A, with a link in region B mirroring A's topics (read-only) plus offset and ACL sync. On a disaster, promote/failover the mirror topics in B and repoint producers/consumers there. Fail-back later means setting up a reverse link from B to A, catching it up, then cutting back.

solid answer

~50 s

Topology: Region A is active; Region B is passive with a cluster link (created in B) mirroring A's topics. Mirror topics in B are read-only; enable consumer-offset sync and ACL sync so positions and authz are ready. On disaster: if A is reachable, **promote** (zero RPO); if A is down, **failover** (accept un-replicated tail loss), then repoint clients' bootstrap servers to B and resume — consumers continue at synced offsets. Fail-back is the hard part: once A recovers you can't just resume the old link (B has diverged with new writes). You stand up a **reverse link from A pulling B**, mirror B's topics into A read-only, drain, then promote back to A and repoint clients again. Key constraints: links are **unidirectional** (no automatic bidirectional active-active), there's no auto-failover (you orchestrate it), offset/ACL sync is **periodic** (small position RPO), schemas/Schema Registry must be replicated separately, exactly-once/transactional state needs care, and network/credentials between regions must be reliable. Plan and drill the runbook.

go deeper

for a junior

Understand the basic shape: active region, passive read-only mirror, manual cutover.

for a middle

Explain failover repointing and that offsets/ACLs can be synced for a clean resume.

for a senior

Reason about promote-vs-failover RPO, periodic sync lag, and that fail-back needs a reverse link.

for a principal

Own the end-to-end DR design: unidirectional constraint, split-brain avoidance, schema/EOS replication, runbook + drills, and cost/networking.

## Goal **Active/passive DR**: one region serves traffic (active), the other stands ready (passive). If the active region fails, you cut over to the passive region with minimal data loss and downtime. ## Topology - **Region A (active)**: receives all producer writes; consumers run here. - **Region B (passive)**: a **cluster link** created in B pulls A's topics into **read-only mirror topics**. Enable: - **consumer offset sync** (`consumer.offset.sync.enable` + group filters) so B knows where each group is, - **ACL sync** (`acl.sync.enable` + filters) so authz is ready, - optionally **auto-create** so new topics in A appear in B. - Because mirroring is **offset-preserving**, B is a faithful, position-accurate copy. ## Failover (A → B) 1. Decide promote vs failover: - **A still reachable** (e.g., planned drill or partial issue) → **promote**: drains mirror lag → **RPO = 0**. - **A is down** → **failover**: immediate writable cutover, **un-replicated tail may be lost** (nonzero RPO). 2. The mirror topics in B become writable (STOPPED state). 3. **Repoint clients**: producers/consumers change `bootstrap.servers` to B. Consumers resume at the **synced offsets** (subject to a small sync lag) with their ACLs already present. 4. There is **no automatic failover** — you (or your automation) execute this runbook. ## Fail-back (B → A) This is where teams get burned. After failover, B has accepted **new writes** that A never saw, so A is now *stale and divergent*. You cannot simply re-enable the old A→B link. Steps: 1. Once A is healthy, create a **reverse link in A** that mirrors B's topics into A (read-only). 2. Let it **catch up** (drain lag) — this can move a lot of data if B ran for a while. 3. During a maintenance window, **promote** the topics back in A and **repoint clients** to A. 4. Re-establish the original A→B link for steady state. ## Key constraints & trade-offs - **Unidirectional links**: a link mirrors one direction. True **active-active** with the same topic written on both sides isn't supported by a single link (offset collisions); you'd use separate topics/prefixes per region. - **No auto-failover / no split-brain protection built in**: you must avoid both regions accepting writes to the same logical topic simultaneously. - **Periodic sync**: offset/ACL sync is not real-time, so there's a small **position RPO** even when data is fully replicated. - **Schema Registry**: schemas are **not** carried by the topic mirror; replicate Schema Registry separately (e.g., Schema Linking) or you'll fail to deserialize after cutover. - **Transactions/EOS**: producer transactional state and `__transaction_state` don't trivially translate; design for at-least-once on failover unless you've validated EOS behavior. - **Networking/security**: cross-region reachability (PrivateLink/peering), credential rotation, and quota/throughput on the link must be sized for the full backlog after an outage. - **Confluent-only & cost**: Cluster Linking is a Confluent feature; cross-region egress and storage cost money. ## RPO/RTO framing - **RPO**: ~0 for promote; the un-replicated tail + position-sync lag for failover. - **RTO**: dominated by detection + the manual/automated cutover + client repointing. ## Bottom line Cluster Linking gives you a low-overhead, offset-accurate passive replica, but DR correctness lives in the **runbook**: promote-vs-failover decisioning, fail-back via a reverse link, schema replication, and split-brain avoidance. Drill it.

  • After failing over to B, why can't you just resume the original A→B link to fail back?
    Because B accepted new writes during the outage, so it has diverged from A. Resuming A→B would try to overwrite/conflict with B's new data. You instead create a reverse link (A pulls B), drain it, then promote back to A.
  • Name two things that are NOT automatically replicated by topic mirroring that you must handle for a clean failover.
    Schema Registry subjects/schemas (use Schema Linking or replicate separately) and producer transactional/exactly-once state. Consumer offsets and ACLs ARE handled, but only if you explicitly enable their sync.
  • Why is naive active-active (same topic written in both regions over one link) problematic?
    Cluster links are unidirectional and offset-preserving; if both sides accept writes to the same logical topic you get offset collisions and split-brain. Active-active requires separate topics/prefixes per region or a different design.

saying these in an interview costs you the question

  • Assuming Cluster Linking auto-fails-over or prevents split-brain
  • Thinking you can fail back by re-enabling the original link
  • Forgetting Schema Registry isn't carried by topic mirroring
  • Believing a single link supports active-active same-topic writes
  • Ignoring that offset/ACL sync is periodic (nonzero position RPO)

context