You put nginx behind a CDN and its existing `limit_req_zone $binary_remote_addr` rule suddenly throttles everyone at once. What is the cause, and how do you correct the rate-limit key without making it spoofable?

seniorshould knowfreq 48%

answer

remote_addr is the TCP peer, not the user
all traffic collapses into a few buckets
trusted proxy ranges must be declared
the raw forwarded header is attacker-controlled
the module rewrites the address before limiting

basics

~20 s

Every request now arrives from a CDN edge address, so all traffic shares one bucket. Fix it with nginx's realip module: set_real_ip_from for the CDN ranges plus real_ip_header, which rewrites $remote_addr before limit_req reads it. Never trust the header from untrusted sources.

solid answer

~50 s

$binary_remote_addr is the address of whoever opened the TCP connection. Once a CDN sits in front, that is an edge node, and a handful of edge addresses now key the whole world's traffic into a handful of buckets — so the first busy second exhausts them and everyone gets 503. The fix is ngx_http_realip_module: list the CDN's ranges with set_real_ip_from, name the header with real_ip_header X-Forwarded-For;, and add real_ip_recursive on; when there are several proxy hops. The module rewrites $remote_addr and $binary_remote_addr early enough that limit_req keys on the real client. The security half matters as much: the module only accepts the header from addresses you listed, so a direct-to-origin request cannot forge it. Keying the zone directly on $http_x_forwarded_for instead is the trap — a client can mint a new value per request and evade the limit entirely, or reuse someone else's and exhaust their bucket.

`$binary_remote_addr` is the address of whoever opened the TCP connection. Once a CDN sits in front, that is an edge node, and a handful of edge addresses now key the whole world's traffic into a handful of buckets — so the first busy second exhausts them and everyone gets 503. The fix is `ngx_http_realip_module`: list the CDN's ranges with `set_real_ip_from`, name the header with `real_ip_header X-Forwarded-For;`, and add `real_ip_recursive on;` when there are several proxy hops. The module rewrites `$remote_addr` and `$binary_remote_addr` early enough that `limit_req` keys on the real client. The security half matters as much: the module only accepts the header from addresses you listed, so a direct-to-origin request cannot forge it. Keying the zone directly on `$http_x_forwarded_for` instead is the trap — a client can mint a new value per request and evade the limit entirely, or reuse someone else's and exhaust their bucket.

code

nginx · 20 lines

nginx

http {
    set_real_ip_from 203.0.113.0/24;
    set_real_ip_from 198.51.100.0/24;
    real_ip_header    X-Forwarded-For;
    real_ip_recursive on;

    limit_req_zone $binary_remote_addr zone=api:10m rate=10r/s;

    log_format main '$remote_addr fwd="$http_x_forwarded_for" "$request" $status';

    server {
        listen 80;
        access_log /var/log/nginx/access.log main;

        location /api/ {
            limit_req zone=api burst=20 nodelay;
            proxy_pass http://backend;
        }
    }
}

go deeper

for a junior

Know that $remote_addr is the address of whatever opened the connection, so behind any proxy it is the proxy. Recognise that a rate limit keyed on it then groups unrelated users together.

for a middle

Name the three realip directives and what each does: set_real_ip_from for trusted peers, real_ip_header for the field, real_ip_recursive for multi-hop chains. Explain that the rewrite happens before limit_req reads the key.

for a senior

Diagnose it from the evidence — broad simultaneous 503s, a log full of one repeating address — and explain both failure modes of trusting the raw header: evasion by fabricating a value, and denial of service by borrowing someone else's.

for a principal

Own the trust boundary as a whole: which tiers may assert a client address, how those ranges stay current as a CDN changes them, whether the origin is reachable around the edge at all, and when identity is a better rate-limit key than address.

What broke

$remote_addr in nginx is the peer address of the TCP connection — nothing more. Direct from the internet, that is the client. Behind a CDN, a cloud load balancer, or another proxy tier, it is that intermediary.

A rate-limit zone keyed on it therefore stops being per client and becomes per proxy node. With rate=10r/s and a CDN presenting, say, twelve edge addresses to your origin, your entire user base is metered into twelve buckets at 10 r/s each. The symptom is unmistakable in hindsight: broad, simultaneous 503s that correlate with total traffic rather than with any individual client, and an error log full of limiting requests, excess: lines naming a small, repeating set of addresses.

The same bug shows up with allow/deny ACLs (they now match the proxy, so an origin allowlist either blocks everyone or nobody), with limit_conn, and in access logs where every line shows the same address.

The realip module

nginx ships ngx_http_realip_module — included in the official packages, and buildable with --with-http_realip_module. It rewrites the client address from a header, but only for connections whose peer you have declared trusted.

nginx

http {
    set_real_ip_from 203.0.113.0/24;   # CDN egress ranges
    set_real_ip_from 198.51.100.0/24;
    real_ip_header   X-Forwarded-For;
    real_ip_recursive on;

    limit_req_zone $binary_remote_addr zone=api:10m rate=10r/s;
}

Three directives, each doing one job:

set_real_ip_from — an address, CIDR range or unix:. Repeat it for every trusted hop. This is the trust anchor and the whole security of the scheme rests on it.
real_ip_header — which header carries the address: X-Forwarded-For, X-Real-IP, any other field name, or proxy_protocol when the front tier speaks the PROXY protocol instead of adding a header.
real_ip_recursive — with off (the default), nginx takes the last address in the forwarded list. With on, it walks the list from the right, skipping addresses that are themselves trusted, and takes the first untrusted one. That is what you want with two or more proxy tiers.

Because the module runs early in request processing, the replacement is visible to limit_req, limit_conn, allow/deny, geo, and the log format — you do not have to change any of them.

Why not just key on the header

The tempting shortcut is:

nginx

# do not do this
limit_req_zone $http_x_forwarded_for zone=api:10m rate=10r/s;

This fails in both directions.

Evasion. The value is attacker-controlled. A client sends a different fabricated address on every request, gets a fresh bucket each time, and is never limited. The zone meanwhile fills with junk keys and evicts the entries of real clients.

Targeted denial of service. The attacker instead sends your address on every request and exhausts your bucket, so you get 503s from traffic you never sent.

Correctness. $http_x_forwarded_for is the raw field: with multiple hops it is a comma-separated list, so the key is a string that changes shape depending on the path a request took.

The realip module avoids all three because it only honours the header from peers in set_real_ip_from. A request that arrives at the origin directly — bypassing the CDN — has an untrusted peer, so its forged header is ignored and it is metered on its true address.

Close the bypass path too

Realip fixes the key; it does not stop someone from skipping the CDN. If your origin is reachable on the public internet, an attacker who learns its address gets the un-cached, un-shielded path. The complement is a network control — a firewall or security group that only admits the CDN's published ranges — or an origin-authentication header the CDN adds and nginx checks. Rate limiting on a bypassable origin is a limit you can be walked around.

Verifying the fix

Put the effective client address in the log format so the change is observable rather than assumed:

nginx

log_format main '$remote_addr fwd="$http_x_forwarded_for" '
                '"$request" $status';

After reload, $remote_addr should show varied client addresses and $http_x_forwarded_for should show the original chain. If $remote_addr still shows one repeating value, either a range is missing from set_real_ip_from or the front tier is sending a different header than the one you named. Both are configuration mistakes with an obvious signature in that one log line.

## What broke `$remote_addr` in nginx is the peer address of the TCP connection — nothing more. Direct from the internet, that is the client. Behind a CDN, a cloud load balancer, or another proxy tier, it is that intermediary. A rate-limit zone keyed on it therefore stops being *per client* and becomes *per proxy node*. With `rate=10r/s` and a CDN presenting, say, twelve edge addresses to your origin, your entire user base is metered into twelve buckets at 10 r/s each. The symptom is unmistakable in hindsight: broad, simultaneous 503s that correlate with total traffic rather than with any individual client, and an error log full of `limiting requests, excess:` lines naming a small, repeating set of addresses. The same bug shows up with `allow`/`deny` ACLs (they now match the proxy, so an origin allowlist either blocks everyone or nobody), with `limit_conn`, and in access logs where every line shows the same address. ## The realip module nginx ships `ngx_http_realip_module` — included in the official packages, and buildable with `--with-http_realip_module`. It rewrites the client address from a header, but only for connections whose peer you have declared trusted. ```nginx http { set_real_ip_from 203.0.113.0/24; # CDN egress ranges set_real_ip_from 198.51.100.0/24; real_ip_header X-Forwarded-For; real_ip_recursive on; limit_req_zone $binary_remote_addr zone=api:10m rate=10r/s; } ``` Three directives, each doing one job: - **`set_real_ip_from`** — an address, CIDR range or `unix:`. Repeat it for every trusted hop. This is the trust anchor and the whole security of the scheme rests on it. - **`real_ip_header`** — which header carries the address: `X-Forwarded-For`, `X-Real-IP`, any other field name, or `proxy_protocol` when the front tier speaks the PROXY protocol instead of adding a header. - **`real_ip_recursive`** — with `off` (the default), nginx takes the last address in the forwarded list. With `on`, it walks the list from the right, skipping addresses that are themselves trusted, and takes the first untrusted one. That is what you want with two or more proxy tiers. Because the module runs early in request processing, the replacement is visible to `limit_req`, `limit_conn`, `allow`/`deny`, `geo`, and the log format — you do not have to change any of them. ## Why not just key on the header The tempting shortcut is: ```nginx # do not do this limit_req_zone $http_x_forwarded_for zone=api:10m rate=10r/s; ``` This fails in both directions. **Evasion.** The value is attacker-controlled. A client sends a different fabricated address on every request, gets a fresh bucket each time, and is never limited. The zone meanwhile fills with junk keys and evicts the entries of real clients. **Targeted denial of service.** The attacker instead sends *your* address on every request and exhausts your bucket, so you get 503s from traffic you never sent. **Correctness.** `$http_x_forwarded_for` is the raw field: with multiple hops it is a comma-separated list, so the key is a string that changes shape depending on the path a request took. The realip module avoids all three because it only honours the header from peers in `set_real_ip_from`. A request that arrives at the origin directly — bypassing the CDN — has an untrusted peer, so its forged header is ignored and it is metered on its true address. ## Close the bypass path too Realip fixes the key; it does not stop someone from skipping the CDN. If your origin is reachable on the public internet, an attacker who learns its address gets the un-cached, un-shielded path. The complement is a network control — a firewall or security group that only admits the CDN's published ranges — or an origin-authentication header the CDN adds and nginx checks. Rate limiting on a bypassable origin is a limit you can be walked around. ## Verifying the fix Put the effective client address in the log format so the change is observable rather than assumed: ```nginx log_format main '$remote_addr fwd="$http_x_forwarded_for" ' '"$request" $status'; ``` After reload, `$remote_addr` should show varied client addresses and `$http_x_forwarded_for` should show the original chain. If `$remote_addr` still shows one repeating value, either a range is missing from `set_real_ip_from` or the front tier is sending a different header than the one you named. Both are configuration mistakes with an obvious signature in that one log line.

What does `real_ip_recursive on;` change when several proxies are chained?
With it off, nginx takes the last address in the forwarded list — which, with two proxy tiers, is your own inner proxy rather than the client. With it on, nginx walks the list from the right and skips any address that is itself covered by `set_real_ip_from`, stopping at the first untrusted one. That is the client as far as your trust boundary can establish it.
You have applied the realip module correctly. Why might the rate limit still be evadable?
Because the origin may still be reachable directly. An attacker who finds the origin address connects past the CDN, and although realip then ignores their forged header, they are also bypassing the CDN's own protections. Restrict the origin at the network layer to the CDN's published ranges, or require a secret origin header the CDN injects and nginx verifies.
The front tier speaks the PROXY protocol rather than adding a header. How does the configuration change?
Set `real_ip_header proxy_protocol;` and add `proxy_protocol` to the relevant `listen` directive so nginx parses the preamble on the connection. `set_real_ip_from` still declares which peers may supply it. The advantage is that the client address travels on the connection itself, which works for TCP streams where there is no HTTP header to carry it.
After the change, why do rate limits look stricter for some corporate users?
Because keying on client address makes a shared NAT egress or corporate proxy a single bucket for many people. That is inherent to address-based keying, not a bug. If it matters, key on an authenticated identity — an API key or tenant extracted with `map` — for logged-in traffic, and keep the address key only for anonymous requests.

saying these in an interview costs you the question

Keying the zone directly on $http_x_forwarded_for
Assuming $remote_addr is always the end user
Trusting the forwarded header from any source
Skipping set_real_ip_from and only setting real_ip_header
Forgetting the origin is still reachable around the CDN

You put nginx behind a CDN and its existing `limit_req_zone $binary_remote_addr` rule suddenly throttles everyone at once. What is the cause, and how do you correct the rate-limit key without making it spoofable?

§ answer

basics

solid answer

code

§ go deeper

▾level notes4 levels

▾deep dive~3 min

What broke

The realip module

Why not just key on the header

Close the bypass path too

Verifying the fix

▾common follow-ups4 questions

▾red flags5 flags

§ context

▾related questions9 questions

▾related topics4 topics

What broke

The realip module

Why not just key on the header

Close the bypass path too

Verifying the fix

answer

go deeper

context