A custom request header your client sets never reaches the application once traffic goes through a CDN and load balancer, though it works when you call the service directly. How do you diagnose and fix that?

## First: prove where it dies Guessing wastes hours. Stand up (or use) an endpoint that returns the request headers it received, then walk the path: 1. curl the **origin** directly, bypassing everything — confirms the app reads it. 2. curl the **internal load balancer / ingress** address. 3. curl through the **CDN / public hostname**. The first step where the field vanishes is the culprit. If the client is a browser, also check the browser devtools network panel: a header can be missing because the *browser* never sent it, which is a completely different problem from a proxy dropping it. ## Cause 1: allowlist forwarding The most common. CDNs, API gateways and some meshes forward a fixed set of headers to the origin (often for cache-key hygiene) and drop the rest, or require you to name extra fields explicitly in an origin request policy. Some also drop unknown headers on the *response* path. Fix: add the field to the forward configuration. Beware the cache-key side effect — if the header varies per user and the CDN includes it in the cache key, you fragment the cache; if it varies per user and is *not* in the key, you risk serving one user's response to another. `Vary` on the response is the correct way to express "the response depends on this field". ## Cause 2: the underscore trap HTTP field names are ASCII tokens where `_` is technically legal, but by default nginx silently discards headers containing underscores (`underscores_in_headers off`), because CGI and many application environments map `-` to `_` when building environment variables, making `X_Foo` and `X-Foo` collide. Symptom: the field survives every other hop and quietly disappears at nginx. Fix: rename to hyphens; do not turn the nginx setting on unless you understand the collision risk. ## Cause 3: hop-by-hop semantics HTTP/1.1 distinguishes **end-to-end** fields from **hop-by-hop** ones. The permanently hop-by-hop set is `Connection`, `Keep-Alive`, `Transfer-Encoding`, `TE`, `Trailer`, `Upgrade`, `Proxy-Authorization`, `Proxy-Authenticate`. Crucially, **`Connection` can name additional fields**: `Connection: X-Acme-Internal` declares that field hop-by-hop, and a conforming proxy MUST strip it before forwarding. If some middleware or client library adds your field to `Connection`, it dies at the first intermediary and the symptom looks exactly like a proxy allowlist. (In HTTP/2 and HTTP/3 the `Connection` header is prohibited entirely, and a message carrying it is malformed — which is another way an intermediary can reject or rewrite traffic.) ## Cause 4: size limits Every server and proxy caps individual field size and total header block size — commonly 4–16 KB. Exceeding it yields **431 Request Header Fields Too Large**, or a 400 from a hop you do not operate, or silent truncation. A large custom header (an encoded token, a serialised context blob) is a classic trigger, especially when combined with big cookies. Fix: shrink it, or move the payload into the body. ## Cause 5: the browser, not the network Any request header outside the CORS-safelisted set makes a cross-origin request **non-simple**, so the browser sends an `OPTIONS` **preflight** first. If the server's `Access-Control-Allow-Headers` does not name your field, the browser blocks the real request — the origin never sees anything, and the network trace shows only an OPTIONS. Mirror image on the response side: a custom response header is present on the wire but unreadable from JavaScript unless the server lists it in `Access-Control-Expose-Headers`. ## Cause 6: name normalisation and duplicates Field names are case-insensitive, and HTTP/2 and HTTP/3 transmit them lowercase; code doing `headers.get("X-Acme-Id")` against a case-sensitive map will "lose" a header that did arrive. Repeated fields with the same name may be combined into one comma-separated value by an intermediary, so code reading only the first occurrence may see a different value than expected. ## The fix, and the lesson Mechanically: configure the dropping hop to forward the field, rename off underscores, keep it small, add it to the CORS allow list, and stop naming it in `Connection`. Architecturally: **any custom header that must survive is part of your infrastructure contract**, not just application code. It belongs in the ingress/CDN configuration that ships with the service, and in a smoke test that asserts it arrives through the real public path — otherwise the next CDN policy change silently breaks it again. Where the data is not intermediary-relevant, ask whether it should be in the request body or URI instead, where nothing strips it.