skip to content

How do replication throttles work during a partition reassignment, and which configs control them?

level: seniorimportance: must knowfreq 60%

answer

  1. rate = how fast, replicas = which
  2. leader rate = source, follower rate = destination
  3. throttled.replicas = partition:broker selector
  4. too-low throttle < produce rate → never catches up
  5. verify deletes throttle

basics

~10 s

Throttles cap how fast replicas copy data during a reassignment so it doesn't starve normal traffic. leader.replication.throttled.rate and follower.replication.throttled.rate (bytes/sec, per broker) set the limit; throttled-replicas lists which replicas are throttled.

solid answer

~40 s

A reassignment moves large amounts of data and can saturate disk/network, hurting live producers/consumers. Throttling caps replication bandwidth. There are broker-level dynamic configs leader.replication.throttled.rate and follower.replication.throttled.rate (bytes/sec) — set on the source/leader brokers and destination/follower brokers respectively. Topic-level configs leader.replication.throttled.replicas and follower.replication.throttled.replicas (lists of partition:broker) mark exactly which replica movements are subject to the throttle, so you don't throttle ordinary replication. Passing --throttle to kafka-reassign-partitions --execute sets all of these automatically; --verify removes them once movement completes. A throttle set too low makes the reassignment crawl (and can even stall if it's below the ongoing produce rate); too high defeats the purpose. You can raise the throttle mid-flight by re-running execute with a new --throttle value.

go deeper

for a junior

Know that a throttle caps replication speed so reassignments don't hurt live traffic.

for a middle

Name the leader/follower rate configs and that --throttle/--verify set/clear them.

for a senior

Explain rate-vs-selector split, the stall when throttle < produce rate, and mid-flight bumping.

for a principal

Reason about choosing throttle values from cluster I/O headroom and SLOs, and monitoring throttle-time metrics during large drains.

## Why throttle at all When a replica is moved to a new broker, the new broker must copy the entire partition log — potentially gigabytes — as fast as it can. Unthrottled, this **replication fetch** competes with live producer writes and consumer reads for disk I/O and network, degrading p99 latency for the whole cluster. A throttle bounds the replication rate so reassignments run as low-priority background work. ## The four configs Kafka splits the throttle into a **rate** (how fast) and a **selector** (which replicas): **Broker-level dynamic configs (bytes/sec):** - `leader.replication.throttled.rate` — caps the rate at which a broker *serves* throttled replication traffic as a leader (the source side sending data). - `follower.replication.throttled.rate` — caps the rate at which a broker *fetches* throttled replication as a follower (the destination side receiving data). These are set per-broker via `kafka-configs.sh --alter --entity-type brokers --entity-name <id>` and are **dynamic** (no restart). **Topic-level configs (selectors):** - `leader.replication.throttled.replicas` — list of `partition:brokerId` entries whose leader side is throttled. - `follower.replication.throttled.replicas` — list whose follower side is throttled. The selector matters: without it, the rate cap would throttle *all* replication for the topic, including normal catch-up after a broker restart. The reassignment tool computes exactly which `partition:broker` pairs are moving and lists only those. ## How --throttle wires it up `kafka-reassign-partitions --execute --throttle 50000000` sets both broker rates to 50 MB/s and populates the two topic replica-lists for every moving replica. `--verify` (after completion) deletes all four. You can use `*` for the replica list to mean 'all replicas'. ## Edge cases and pitfalls - **Too-low throttle deadlock-ish stall:** if the throttle is below the topic's incoming produce rate, the moving replica can never catch up to join the ISR — the reassignment never completes. Throttle must exceed produce throughput. - **Forgetting --verify** leaves throttle configs in place, silently slowing future replication. - **Mid-flight bump:** re-running `--execute` with a higher `--throttle` updates the rate live. - Metric to watch: `kafka.server:type=...,name=...ThrottleTime` and per-broker replication MB/s.

  • What happens if you set the throttle lower than the topic's produce rate?
    The moving replica can never catch up to the leader, so it never joins the ISR and the reassignment stalls indefinitely. The throttle must exceed incoming throughput.
  • Why are there separate leader and follower throttle rates?
    The leader (source) side caps outbound serving of throttled replication; the follower (destination) side caps inbound fetching. They sit on different brokers and limit different directions of the same transfer.

saying these in an interview costs you the question

  • Saying the throttle is a single global config — there are distinct leader/follower rates plus topic-level replica selectors.
  • Claiming throttles apply to all replication automatically — the throttled.replicas selector limits it to the moving replicas.
  • Not knowing that an under-set throttle can stall the reassignment forever.

context

resolving links…