An nginx `upstream` block uses `ip_hash;` and one backend receives most of the traffic from a large corporate customer while the others sit idle. What exactly does nginx hash, and why does that produce the skew?

## What the directive does ```nginx upstream backend { ip_hash; server 10.0.0.11:8080; server 10.0.0.12:8080; server 10.0.0.13:8080; } ``` With `ip_hash`, nginx derives a key from the client address and uses it to select a server, so requests from that address consistently reach the same backend. The documented key is the **first three octets of an IPv4 address**, or the **entire IPv6 address**. That IPv4 detail is the whole answer to the skew question: addresses within the same /24 hash to the same value. ## Why one backend gets the traffic Address-based mapping assumes addresses are distributed roughly like users. Real networks break that assumption comprehensively: - A corporate office of thousands of employees leaves through one or a few NAT addresses. - Mobile carriers place enormous user populations behind CGNAT ranges. - An automated client, scraper or partner integration is one address making a large share of the requests. Any of those is a single hash key. Nginx is behaving exactly as specified; the distribution is uneven because the input is uneven. Adding backends does not help — the heavy key still resolves to exactly one of them. ## The second trap: changing the server set The mapping is computed over the live servers in the group. Remove one and a substantial share of clients are remapped to different backends, losing whatever local state made affinity worth having in the first place. Nginx provides `down` for this: ```nginx upstream backend { ip_hash; server 10.0.0.11:8080; server 10.0.0.12:8080 down; server 10.0.0.13:8080; } ``` Marking a server `down` takes it out of rotation while preserving the current hashing of client addresses for the rest, which is what you want during maintenance. Scaling the group up or down is a different matter: the reshuffle is inherent, and it is why affinity and elastic capacity pull against each other. ## The generic hash directive When you want affinity keyed on something other than the network address, the `hash` directive takes an arbitrary key built from variables, and the optional `consistent` parameter switches to a ketama-style ring so that adding or removing a server remaps far fewer keys: ```nginx upstream backend { hash $cookie_sessionid consistent; server 10.0.0.11:8080; server 10.0.0.12:8080; } ``` Hashing a session cookie distributes per user rather than per network, which fixes the NAT skew directly, at the cost of requiring that the key exists on every request — requests without the cookie fall back to hashing an empty key and pile onto one server, so make sure the key is always present. ## Realip changes the input If nginx sits behind a CDN or another load balancer, the peer address is that hop's. The realip module rewrites the peer address early in request processing, so once `set_real_ip_from` and `real_ip_header` are configured, `ip_hash` keys off the recovered client address rather than the front hop's. Without that configuration the symptom is extreme: every request shares a handful of addresses and effectively one backend receives everything. ## Ordering inside the block One mechanical rule that trips people: when a group uses a balancing method other than the default round robin, the method directive must appear **before** `keepalive` in the block. Config that mixes `ip_hash` and upstream keepalive in the wrong order does not behave as intended. ## What to say in an interview Name the key precisely — three octets for IPv4, the full address for IPv6 — then connect it to NAT and CGNAT as the cause of the skew, then mention the two operational consequences: `down` for preserving the map during maintenance, and a cookie-based `hash ... consistent` when you need per-user rather than per-network distribution. That sequence shows you have read the directive's behaviour rather than assumed it hashes the full address.