Load Balancing

## What a load balancer is and why it exists A load balancer is the component that sits between clients and a fleet of backend servers, accepting incoming connections or requests and deciding, per request, which backend instance actually handles it. It exists to solve two problems that appear the moment an application outgrows a single machine. 1. **The first is capacity:** a single server has a ceiling on CPU, memory, and network throughput, so once traffic exceeds that ceiling you must run several copies of the service and spread traffic across them. 2. **The second is availability:** if a client talks directly to one server and that server crashes or is redeployed, every one of that client's requests fails; a load balancer can detect a dead instance and simply stop sending it traffic, so the failure of one box does not become an outage for the whole service. Underneath, a load balancer maintains a **pool** (sometimes called an upstream or target group) of backend addresses and applies an algorithm to pick one for each new request or connection. ## Round-robin: taking turns Round-robin is the simplest algorithm: the balancer keeps a pointer into the pool and advances it after every request, so `server1` gets request 1, `server2` gets request 2, and so on, wrapping back to `server1` after the last entry. - **Cheap, and even in request count.** It requires no runtime state about the servers themselves beyond the pool membership, which makes it cheap to implement and easy to reason about, and it delivers a genuinely even distribution of request *count* over time. - **Blind to actual load.** Its weakness is that it distributes requests, not work. If request costs vary, one server can end up doing far more CPU-seconds of work than another purely by bad luck of ordering, and if one server is already bogged down handling a slow request, round-robin will still send it the next one in strict turn. ## Least-connections: deciding on live load Least-connections fixes exactly that blind spot by making the decision load-aware: the balancer tracks how many requests (or, for L4 balancers, how many open connections) each backend currently has in flight and always forwards the next request to whichever backend has the fewest. This means: - a backend that is stuck processing a slow query naturally receives fewer new requests until it catches up; - a backend that just finished a batch of fast requests gets more work sooner. This **self-correcting** property makes least-connections noticeably better than round-robin whenever request duration is heterogeneous - which describes almost all real web and API traffic, where some endpoints are a cache hit and others trigger a multi-table join or an outbound call to a third party. The cost is a small amount of bookkeeping: the balancer (or, in a distributed setup, each balancer node) must maintain a live counter per backend and update it on request start and completion, which is trivial for a single load balancer process but requires coordination or approximation when load is split across many balancer instances that do not share state. ## Weighted variants for uneven fleets A closely related refinement is weighted variants of both algorithms. | Variant | How the weight is applied | |---|---| | **weighted round-robin** | gives faster or bigger instances a proportionally larger share of the turns | | **weighted least-connections** | divides the connection count by a capacity weight before comparing, so a server rated at twice the capacity is allowed roughly twice the concurrent connections before it stops receiving new ones | This matters in heterogeneous fleets - for example during a rolling deploy where new instances are still warming up caches and JIT-compiling hot paths, or in mixed-instance-type autoscaling groups on AWS where a `c5.xlarge` and a `c5.2xlarge` sit in the same target group. Real load balancers such as **NGINX**, **HAProxy**, and AWS's Application/Network Load Balancers default to round-robin or a randomized-choice variant for simplicity but expose least-connections and weighted modes as configuration, and HAProxy in particular also supports a hybrid 'first' policy and consistent-hashing-based balancing for cache-affinity use cases. The engineering decision in practice is rarely 'round-robin vs least-connections' in the abstract; it is choosing the algorithm that matches the traffic's variance in per-request cost, and re-checking that choice whenever the workload mix changes, for instance when a new expensive endpoint is added to a service that was previously uniform. ## A subtler option: least-response-time There is also a subtler algorithm worth naming: least-response-time (sometimes called weighted-response-time), which factors in not just the current connection count but each backend's recent observed latency, favoring instances that are both lightly loaded and currently fast. This catches a case plain least-connections misses: - a backend can have a low connection count simply because it is slow and requests are backing up elsewhere; - conversely a backend can look 'busy' by connection count while actually being extremely fast per request. Response-time-aware balancing needs the load balancer to track a rolling latency metric per backend, which is more bookkeeping than a raw counter but produces routing decisions that track real-world performance more closely. ## Every algorithm has edge cases None of these algorithms are free of edge cases: any algorithm that reacts to a live signal (connection count, latency) can overreact to noise, and any algorithm that ignores load (plain round-robin) can systematically starve or flood a backend under skewed traffic patterns, which is why production teams typically pick the simplest algorithm that actually matches their traffic's characteristics rather than reaching for the most sophisticated option by default.