After recovering the original primary, how do you fail back (resync) to it in an active-passive setup, and why is unplanned-failover fallback the hard case?

## What fallback (failback / resync) means After a failover, the **standby** is now serving as the active cluster. **Fallback** is returning to the original primary once it's healthy: you reverse the replication direction so the recovered primary is brought up to date from the current active cluster, then perform a controlled cutover back. It's effectively "a planned failover, in the opposite direction." ## Clean case: fallback after a planned failover Because the original failover was **planned** — producers quiesced, replication drained to lag 0 — the two clusters were **consistent** at the moment of cutover. The old primary holds exactly the data the standby had. So: 1. Configure replication **standby → old-primary** (a new MM2 flow or reverse Cluster Link). 2. Let it copy everything produced on the standby since cutover; wait for lag 0. 3. Re-establish **offset translation** in the new direction (checkpoints standby → old-primary). 4. Do a planned failover back: quiesce, drain, translate offsets, repoint clients. No divergence, so this is mechanically just another planned cutover. ## Hard case: fallback after an unplanned failover When the primary died abruptly, it had **un-replicated records** — a **divergent tail** of writes it accepted that never reached the standby (the RPO loss). When the old primary comes back, its logs still contain those records, and they conflict with the standby's now-authoritative timeline: - The same offsets/partitions exist on both clusters with **different content** beyond the divergence point. - Naively replicating standby → old-primary would **append on top of** the stale tail, producing **duplicates and offset/content conflicts**, and can corrupt **log-compacted** topics or **transactional/EOS** state. ### Reconciliation strategies 1. **Rebuild from scratch (most common)**: declare the standby the **single source of truth**, **wipe** the old primary's topics (or the whole cluster), and **re-replicate fully** from the standby. The stranded tail is **discarded** — acceptable when that data was already counted as RPO loss. 2. **Application-level merge**: if the lost records are business-critical (e.g. payments), extract the divergent tail from the recovered primary and **replay it through the application** so the active cluster can absorb it idempotently. This is bespoke and risky; only do it when the data truly matters. 3. **Forensic discard with audit**: keep the old primary's tail as an offline forensic copy, but don't merge — used where regulators require evidence but the system can't reconcile automatically. ## Operational guardrails - **Fence the old primary** the entire time it's recovering so it never accepts client traffic and never becomes a second active cluster (split-brain). - **Re-create internal MM2 topics** for the new direction (offset-syncs, checkpoints) so translation works the right way. - **Validate with a drill**: do the fallback in a maintenance window and verify consumer positions, lag, and data integrity before repointing production clients. - **Document the decision** (discard vs merge) — it's a business call about the RPO-lost data, not purely technical. ## Why this is a principal-level concern Fallback ties together replication direction, offset translation, divergence reconciliation, split-brain prevention, and a business decision about lost data. The technically tempting "just reverse replication" silently corrupts data after an unplanned failover. The disciplined answer is: treat the surviving active cluster as source of truth, rebuild the recovered one, and only do application-level merge when the lost tail is worth the bespoke risk.