Preferred Leader Election and Partition Balancing

## First principles A Kafka **topic** is split into **partitions**. Each partition is replicated onto several brokers for fault tolerance. The list of brokers holding a partition's replicas is called the **assigned replicas (AR)** — and crucially it is *ordered*. For example partition 0 might have AR = [3, 1, 2], meaning broker 3 holds the first replica, broker 1 the second, broker 2 the third. At any moment exactly one replica is the **leader**: it is the only replica that serves producer writes and consumer reads for that partition. The others are **followers** that just copy the leader's log. The subset of replicas that are fully caught up is the **in-sync replica set (ISR)**. ## What 'preferred leader' means The **preferred leader** is simply **the first broker in the AR list** (broker 3 in the example). It is 'preferred' because Kafka assigns AR lists round-robin at topic-creation time, so if every partition's leader were its first replica, leadership would be spread evenly across all brokers. ## Why it matters Because only the leader does I/O for a partition, where the leaders sit determines load distribution. Two things push leadership away from the preferred replica: 1. **Broker restart / failure**: when the preferred leader broker goes down, leadership moves to another in-sync replica. When the broker comes back it rejoins as a *follower*, not automatically as leader. 2. **Repeated churn**: over many restarts, leaders pile up on the brokers that stayed alive. The result is **leader imbalance**: some brokers host far more leaders than others, so they saturate on CPU/network while others idle. Restoring leadership to the preferred replicas re-balances the load. A replica is only eligible to become leader if it is currently in the ISR — an out-of-sync preferred replica is skipped until it catches up. ## How it's triggered - Automatically, if `auto.leader.rebalance.enable=true` (the default), the controller periodically checks imbalance and runs preferred election. - Manually, via `kafka-leader-election.sh --election-type PREFERRED`. ## Edge cases - If the preferred replica is not in the ISR, the election does nothing for that partition (it won't elect an out-of-sync replica unless unclean election is allowed, which is a separate, dangerous setting). - Preferred election only *moves leadership*; it never moves data or changes the replica assignment. Moving replicas requires reassignment.