A round-robin load balancer sends each backend the same number of requests, yet one backend sits at 90% CPU while another is nearly idle. Explain how that happens, and what least-connection and weighted balancing each change.

## What round-robin actually promises Round-robin walks the backend list in order and hands each new request to the next entry. Its guarantee is exact and narrow: over time every backend receives the same *number* of requests. Nothing in the algorithm looks at CPU, memory, queue depth, response time, or whether the previous request to that backend has even finished. If every request cost the same and every backend were identical, count-fairness would equal load-fairness. Neither assumption survives contact with production. ## Four ways equal counts become unequal load **Request cost variance.** A single endpoint mix can span four orders of magnitude — a cached lookup at one millisecond, a report query at ten seconds. Round-robin deals them out like cards, so over a short window one backend can accumulate several expensive requests purely by arrival order. The larger the variance in cost, the longer the window over which the imbalance persists. **Long-lived connections.** At the connection level — WebSockets, gRPC streams, server-sent events, or plain HTTP keep-alive when the balancer routes per connection rather than per request — the balancing decision is made once and then lasts for the connection's lifetime. Backends restarted most recently hold the fewest long-lived connections, and a backend that has been up longest slowly accretes them. This is why a fleet can look badly skewed after a rolling restart even under perfect round-robin. **Heterogeneous capacity.** Mixed instance types, mixed generations of hardware, or containers with different CPU limits mean equal request counts land on unequal machines. **Noisy neighbours and local state.** A backend sharing a host with a busy tenant, or one whose cache is cold after a restart, does the same work more slowly. Its own throughput drops while its share of arrivals does not. ## What least-connection changes Least-connection (sometimes least-request) picks the backend with the fewest requests currently outstanding from this balancer. In-flight count is a proxy for how busy a backend is: a backend that is slow to finish work accumulates outstanding requests and is automatically passed over until it drains. That makes it self-correcting in exactly the cases round-robin is blind to — variable request cost, temporary slowness, cold caches, uneven long-lived connections. The signal has two important limits. First, the count is *local to this balancer*. With several proxies in front of one pool, each sees only its own share, so the global picture can still be lopsided. Second, and more dangerous, **fast failure looks like idleness**: a backend that immediately returns 500s or refuses to do work completes requests instantly, so its in-flight count stays at zero and least-connection preferentially feeds it. This is the classic black-hole pattern, and the mitigation is not in the algorithm — it lives in the health-checking and outlier-ejection layer that removes such a backend from the pool. ## What weighting changes, and what it does not Weighted round-robin (or weighted least-connection) attaches a static integer to each backend and deals proportionally: weight 3 receives three requests for every one that weight 1 receives. This is the right tool for *known, stable* differences — a pool where half the machines have twice the cores, or a deliberate traffic split while you validate a new instance size. It is the wrong tool for anything dynamic, because the weight is a number you decided in advance and traffic changes hourly. Weights also age badly: they are typically set once during an incident and then quietly outlive the reason for them. ## How to choose For homogeneous backends serving short, uniform-cost requests, round-robin is entirely adequate and has the cheapest implementation (no per-backend state, no coordination). Reach for least-connection when request durations vary widely, when a backend can be temporarily slow rather than down, or when connections are long-lived. Use weights only for capacity ratios you can state and would notice going stale. And regardless of algorithm, remember what balancing cannot do: it distributes work among backends that are in the pool, so a backend that should not be receiving traffic at all is a health-check problem, not a balancing one. ## Confirming the diagnosis Do not argue from the algorithm — measure. Compare per-backend request counts against per-backend CPU and per-backend p99 latency. Equal counts with skewed CPU is the cost-variance or capacity story. Skewed counts under round-robin points at something else entirely: connection-level pinning, affinity, or backends that were out of the pool for part of the window.