How do you throttle replication bandwidth in Kafka, and what's the tradeoff between intra-cluster replica throttling and cross-cluster MirrorMaker 2 throttling?

**Two different problems, two mechanisms.** People conflate 'replication throttling' but Kafka has two unrelated layers. **1. Intra-cluster replica throttling (within one cluster).** Inside a single Kafka cluster, followers replicate partitions from leaders. During a *partition reassignment* (moving replicas between brokers, e.g. after adding brokers) this catch-up traffic can flood the network and starve normal producers/consumers. Kafka lets you cap it: - Broker/topic dynamic configs `leader.replication.throttled.rate` and `follower.replication.throttled.rate` (bytes/sec) limit, respectively, how fast a leader serves throttled replicas and how fast a follower fetches. - The topic configs `leader.replication.throttled.replicas` and `follower.replication.throttled.replicas` say *which* replicas the throttle applies to. - In practice you don't hand-set these — `kafka-reassign-partitions.sh --throttle <bytesPerSec>` applies them for the duration of a reassignment and you remove them with `--verify` afterward. You can also set them directly via `kafka-configs.sh --alter --add-config`. This is about protecting *live traffic from rebalancing traffic* inside one cluster. It has nothing to do with cross-cluster mirroring. **2. Cross-cluster MirrorMaker 2 throttling.** MM2 copies data *between* clusters and runs on Kafka Connect. To MM2's source and target clusters it is just a *consumer* (reading the source) and a *producer* (writing the target). So you throttle it with the same tools you throttle any client: - **Client/user quotas**: `producer_byte_rate` and `consumer_byte_rate` (bytes/sec) applied per `client-id`, `user` (principal), or both, via `kafka-configs.sh --alter --add-config 'producer_byte_rate=...,consumer_byte_rate=...' --entity-type clients/users`. Give MM2's Connect workers a known client-id/principal and quota that. - **Parallelism**: `tasks.max` on the MM2 connectors plus source partition count cap how many parallel streams run. - Network-level shaping (QoS on the WAN link) is sometimes layered on top. This is about protecting *the WAN link and the target cluster* from mirror traffic — and conversely, ensuring mirror traffic gets *enough* bandwidth. **The central tradeoff.** Cross-cluster bandwidth is finite (and over a WAN, expensive and shared). - **Throttle too low** -> replication throughput < ingest rate -> **replication lag accumulates** -> your DR copy falls further behind -> **RPO blows out** (you'd lose more data on failover). After any outage the backlog can never drain. - **Throttle too high / none** -> mirror traffic saturates the inter-region link or overloads target brokers -> live produce/consume latency spikes, or other tenants on a shared link suffer. **Doing it right.** - Size the MM2 quota *above peak steady-state ingest* with headroom so lag stays near zero in normal operation. - Keep a *catch-up* plan: after MM2 downtime, temporarily raise the quota (or `tasks.max`) so the backlog drains within the RPO budget, then return to baseline. - Prioritize DR-critical topics; exclude or deprioritize bulk/non-critical topics so the budget protects what matters. - Monitor replication-latency and the lag derivative; an upward trend means the throttle is below ingest. **Edge cases / pitfalls.** Forgetting to remove an intra-cluster reassignment throttle (`--verify` step) leaves replicas permanently rate-limited, risking under-replication. Quotas applied to the wrong client-id miss MM2 entirely. A single global quota across many MM2 tasks can serialize them; quota granularity (per client-id vs per user) matters. Setting the cross-cluster quota *at* ingest rate (no headroom) guarantees creeping lag.