kube-controller-manager runs on all three control-plane nodes of a highly available cluster. Why don't three copies of the ReplicaSet controller each create their own set of Pods?

## Why singleton control loops Controllers are convergent but not commutative in the short term. Two ReplicaSet controllers observing `replicas: 3` with zero Pods would each create three Pods; the next sync would delete the surplus, but you would have churned six Pods and disrupted workloads. Rather than make every loop safe for concurrent actors, Kubernetes makes the loops **singleton**: one active instance cluster-wide per controller-manager (and per scheduler). ## The lease mechanism Leader election is built on ordinary API objects, not a separate consensus service — etcd's Raft already provides the linearizable store, so election reduces to a compare-and-swap on one object. - The lock is a **coordination.k8s.io/v1 Lease** in `kube-system`, named after the component. Historically Endpoints or ConfigMap annotations were used; Lease is the modern default because it is cheap to write and does not pollute Endpoints. - Its fields: `holderIdentity` (usually hostname plus a random suffix), `leaseDurationSeconds`, `acquireTime`, `renewTime`, `leaseTransitions`. - **Acquire**: if the lease is absent or `renewTime` is older than the lease duration, a candidate writes itself as holder. The write carries the observed `resourceVersion`, so the API server rejects all but one concurrent attempt with a conflict — that is the mutual exclusion. - **Renew**: the holder rewrites `renewTime` every retry period. If it cannot renew before the renew deadline (API server unreachable, GC pause, node freeze), it must assume it lost leadership and stop. Relevant flags: `--leader-elect` (on by default), `--leader-elect-lease-duration` (15s), `--leader-elect-renew-deadline` (10s), `--leader-elect-retry-period` (2s). The invariant is renewDeadline < leaseDuration, with enough margin that a slow but healthy leader is not displaced. ## What failover looks like Kill the leader and nothing visible happens for a few seconds: no reconciliation, so a Pod that dies is not replaced, a rollout pauses, a NotReady node is not processed. Once the lease expires a standby acquires it, builds its informer caches from a **list-then-watch** against the API server, and resumes. Because reconciliation is level-triggered, the new leader needs no handoff state — it reads the world and computes the delta. Nothing is lost; work is only delayed. `leaseTransitions` increments, and you can watch the transition with `kubectl get lease -n kube-system kube-controller-manager -o yaml` or the component's `leader_election_master_status` metric. ## Split brain and why it is survivable Leases are time-based, and clocks and pauses are imperfect: a leader stalled by a long GC pause may believe it still holds a lease that has already been taken. Brief overlap is therefore possible. The system tolerates it because: - Every write goes through the API server with **optimistic concurrency** — a stale actor's update fails on resourceVersion conflict. - Controllers are **idempotent and level-triggered** — the worst outcome is extra churn (a surplus Pod created then deleted), not corrupted state. - The stale leader's renewal is failing, so it exits within the renew deadline. This is deliberately weaker than a fencing-token design; Kubernetes trades strict exclusion for simplicity, relying on convergence to clean up. ## Contrast with other components - **kube-apiserver**: stateless and horizontally scalable — all replicas serve traffic behind a load balancer, no election. - **etcd**: has its own Raft leader, unrelated to Kubernetes leases; do not conflate them. - **kube-scheduler**: same lease pattern as controller-manager, its own Lease object. - **kubelet**: no election — one per node, each owning only its own Pods. - **Custom controllers/operators**: use the same client-go leader-election library, and should, for exactly the same reason. Running two operator replicas without it is a real production bug. ## Operating notes Symptoms of election trouble: rapidly incrementing `leaseTransitions` (flapping, usually from an overloaded or high-latency API server) — reconciliation then stalls repeatedly. Extending lease duration reduces flapping but lengthens failover. On single-node control planes leader election still runs; it is a no-op contest with one candidate, and disabling it saves a trivial amount of write traffic but removes the safety net if a second instance ever appears.