How do RPO and RTO targets shape an active-passive Kafka DR design, and what drives each?

## Two numbers that define a DR plan **RPO — Recovery Point Objective**: the maximum amount of data, measured in time, that the business will tolerate losing. RPO = 0 means "lose nothing"; RPO = 5 minutes means "losing the last 5 minutes of events is acceptable." **RTO — Recovery Time Objective**: the maximum tolerable time to restore service after a disaster. RTO = 0 means "no downtime"; RTO = 1 hour means "we can be down up to an hour." These are **business** targets that drive **technical** design and cost. ## What drives RPO in active-passive Kafka Cross-cluster replication in Kafka is **asynchronous**: the primary acknowledges a producer's write based on its own in-sync replicas, and a separate process (MirrorMaker 2, Confluent Replicator, Cluster Linking) later copies those records to the standby. At any instant there is a **replication lag** — records committed on the primary but not yet present on the standby. If the primary is lost abruptly, exactly those records are gone. Therefore: > **RPO ≈ replication lag at the moment of failure.** Levers to reduce RPO: - Increase replication parallelism (MM2 `tasks.max`, more partitions). - Reduce inter-region network latency / increase bandwidth. - Keep producers from out-running replication during bursts. - Monitor lag continuously (MM2 emits `replication-latency-ms` and record-lag metrics; Cluster Linking exposes mirror lag). You **cannot** reach RPO = 0 with async replication. Synchronous cross-region replication would require acknowledging every write on the remote region, adding cross-region round-trip latency to every produce — generally unacceptable, which is why Kafka DR is async. ## What drives RTO RTO is the sum of: 1. **Detection** — how fast monitoring/alerting notices the primary is down. 2. **Decision** — automated vs human-in-the-loop authorization to fail over. 3. **Cutover mechanics** — repointing clients (DNS/bootstrap), translating consumer offsets so consumers resume at the right place, and ensuring the standby has capacity. Standby "temperature" sets the floor: - **Cold standby**: cluster provisioned/scaled up only when needed → minutes-to-hours RTO, lowest cost. - **Warm standby**: cluster already running and continuously replicating → minutes RTO. This is the typical Kafka DR choice. - ("Hot"/active-active gives near-zero RTO but is a different topology.) ## The trade-off triangle Tighter RPO and RTO cost more (bandwidth, idle warm capacity, automation). DR design is choosing concrete numbers the business signs off on — e.g. **RPO 30s, RTO 15min** — then proving them with **regular failover drills**. Numbers you never test are fiction. ## Edge cases - During a **bursty** producer spike, lag (and thus potential RPO loss) grows even if average lag is tiny — size for peak, not average. - **Planned** failover can achieve RPO ≈ 0 by stopping producers and draining replication before cutover; **unplanned** failover cannot. - RTO must include the time to **translate offsets** and restart consumers, not just to point them at a new bootstrap server.