skip to content

A newly added Go service starts a fresh trace instead of continuing the caller's X-Trace-Id. How do you find where the header is lost?

level: seniorimportance: nice to knowfreq 32%

answer

  1. two sides, one wire
  2. eliminate half before guessing
  3. print what arrived, not what you expected
  4. headers only, never the body
  5. a fresh request starts with an empty header map

basics

~20 s

Split the chain at the process boundary first. Log httputil.DumpRequest in the receiving handler to see which header keys arrived: if X-Trace-Id is absent there, the sender or a proxy dropped it; if present, the reading code is wrong.

solid answer

~50 s

Decide which side is broken before changing anything. In the receiving handler I dump the inbound request with `httputil.DumpRequest(r, false)` and log it — that shows the header keys as `net/http` parsed them. If `X-Trace-Id` is in that dump, the bug is on the reading side: usually a raw map index with non-canonical casing where `Header.Get` was needed, or a lookup against a request that was rebuilt without the header. If it is absent, I move to the sender and use `httputil.DumpRequestOut` just before `client.Do`, which shows what `net/http` will actually write. The usual causes there are an outbound request built with `http.NewRequest`, so whatever fills the header from `req.Context()` found nothing to write, and a request rebuilt for a retry without copying the header. If the sender's dump has it and the receiver's does not, something between them is stripping it.

code

go · 8 lines
go
func debugHeaders(next http.Handler) http.Handler {
	return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		if dump, err := httputil.DumpRequest(r, false); err == nil {
			log.Printf("inbound:\n%s", dump)
		}
		next.ServeHTTP(w, r)
	})
}

go deeper

for a junior

Know the two tools by name and which side each belongs on: one dumps a request a server received, the other a request a client is about to send. Passing false for the body is the safe default.

for a middle

Explain what each outcome of the inbound dump proves, and name the two Go-specific ways a sender fails to write the header: a request built without the caller's context, and a request constructed fresh without copying it.

for a senior

Show the bisection discipline — split at the boundary, eliminate a side, only then hypothesise — and handle the safety question about logging headers in production without being prompted.

for a principal

Argue for the fix that outlives the incident: one place that writes propagation headers, one that reads them, and a boundary test in every new service, versus relying on reviewers to notice a missing line.

### Why the first move is a boundary, not a hypothesis A broken chain has exactly two possible locations — before the wire or after it — and one cheap observation separates them. Every minute spent reasoning about which of six plausible Go mistakes it might be is a minute you could have spent eliminating half of them. Dump first, hypothesise second. ### Observing the receiving side ```go dump, err := httputil.DumpRequest(r, false) if err == nil { log.Printf("inbound headers:\n%s", dump) } ``` `DumpRequest` is the server-side tool: it reconstructs the request in HTTP/1 wire form from what the handler holds. Pass `false` for the body — you want header keys, not payload. Two details matter when you read the output. First, the keys shown are canonical, because that is how the parser stored them; the dump cannot tell you the original casing on the wire, and it does not need to. Second, the dump lists *everything* that arrived, which is the point: a `Get` returning `""` cannot distinguish "the header was never sent" from "the sender sent a different name than we agreed on", and a surprising number of these bugs turn out to be the second. The client-side counterpart is `httputil.DumpRequestOut`, which shows the request as the transport will write it, including headers the transport adds itself. Use it on the sending side, not the receiving one. ### Reading the two outcomes **The header is in the inbound dump.** The wire is fine and the reading code is wrong. In order of likelihood: the handler indexes `r.Header["x-trace-id"]` instead of calling `r.Header.Get`, and gets `nil` because the map key is `X-Trace-Id`; the near-miss spelling `X-Trace-ID` was used for the same index; the value is read from a request that some middleware rebuilt without copying the header across; or the read happens on a path that is not the one the request actually took. **The header is not in the inbound dump.** Move upstream and dump there. Two Go-specific causes dominate. One: the outbound request was built with `http.NewRequest`, so it carries an empty root context, and whatever code was supposed to read per-request state out of `req.Context()` and turn it into a header found nothing and wrote nothing. Two: the outbound request was constructed fresh — for a retry, for a fan-out, in a helper written last week — and nobody carried the header onto it. A fresh `*http.Request` starts with an empty header map; reading a header on the inbound request does **not** forward it. Every hop must explicitly write it out again. **Both dumps disagree.** The sender writes it, the receiver does not see it, and the loss is between them: a gateway, a proxy or a mesh that drops header names it does not recognise. This is the one outcome the code cannot fix, and it is worth confirming rather than assuming, because it is the easiest to blame and the hardest to prove without both dumps. ### Narrowing further If the receiving service is behind several hops, dump at each one rather than only at the ends — a chain of five services has four boundaries and you can bisect them. If the header appears intermittently, look for a code path that builds the request differently: a retry path, a cached client, a background refresh, a fan-out that reuses a template request. Intermittency in this class of bug is almost always two construction sites for the same call, one of them missing a line. ### Keeping the instrumentation safe A header dump contains authorisation and cookie values. Do not log it unconditionally in production: pass `false` for the body, redact or delete the sensitive keys from a cloned header before formatting, and put the whole thing behind a debug flag or a sampled path. What starts as a five-minute diagnostic has a way of becoming a permanent line on the hot path that quietly writes credentials into a log aggregator. ### The durable fix Once you have found it, the repair is rarely just the missing line. It is making the missing line impossible: one helper that builds outbound requests and sets the propagation headers, one accessor that reads them, and a test at the boundary of the new service asserting that an inbound header comes back out on the outbound call. A chain that is correct only because every author remembered is a chain that will break again at the next hop somebody adds.

  • Why dump the whole request instead of just logging Header.Get for the key you expect?
    Because an empty `Get` answers the wrong question. It tells you your key is not there, not whether anything was. The dump shows every key that arrived, which catches the case where the sender is setting a differently named header than the one you agreed on — a surprisingly common cause, and invisible to a targeted `Get`.
  • The dump shows the header arriving and the handler reads it fine, but the next hop still gets nothing. Where do you look?
    At the code that builds the outbound request. Reading a header does not forward it: a fresh `*http.Request` starts with an empty header map, so the value has to be written onto the new request explicitly. Check also that the outbound request is built from the handler's context, since code that fills the header often reads the value from there.
  • How do you keep this diagnostic from becoming a production problem?
    Dump headers only, never the body; strip or redact authorisation and cookie keys before logging; and gate it behind a debug flag or a sampled path rather than running it on every request. A permanent version belongs in a debug-only handler, not as an unconditional log line writing credentials into your log store.

saying these in an interview costs you the question

  • Guesses at a cause without observing what arrived
  • Assumes a header that was read is automatically forwarded
  • Dumps full request bodies into production logs
  • Blames the tracing backend before checking the wire
  • Inspects only the sender and never the receiver