Tracing and Sampling

A trace ID (often also called a correlation ID or request ID) is a single opaque value that identifies one logical request and is stamped onto every log record produced while handling it, in every process it touches. It is the cheapest correlation mechanism in observability and the one that most reliably pays for itself. ## What the identifier changes about a log search Without it, investigating one request is reconstruction work. You narrow to a time window, then to a service, then guess at a discriminator that somebody hopefully logged — an account number, an order reference, a URL path. Every filter is a hypothesis, and each one silently drops lines that phrased the same fact differently. On a busy service a two-minute window can hold six figures of requests, so the window alone is nearly useless. With the ID present, the search becomes an equality predicate on a single field. Two properties do the work: - **It is global.** Generated once, unchanged across every hop, so the same value selects lines in the gateway, the service that failed, and the three services behind it. - **It is opaque and effectively unique.** That makes it a perfect log field and a terrible metric dimension — putting it on a metric would create one series per request. Correlation identifiers belong on logs and spans, never as a metric label. ## Where the identifier comes from 1. **Generated** at the first component that touches the request — an edge proxy, an API gateway, or the first instrumented service — if the incoming request does not already carry one. 2. **Propagated** across process boundaries: a header on an HTTP call, a record header or message attribute on a broker, a field on an RPC envelope. 3. **Held** inside the process in a context-local store bound to the unit of work, so any code on that path can read it without it being threaded through every method signature. 4. **Copied** out of that store into each log record at write time, by the logging setup. Step 4 is the one that surprises people. A tracing library knowing the ID is not the same thing as the logs carrying it; something has to bridge the two, and if nobody wired that bridge, traces and logs stay two disconnected worlds. ## The three ways it goes missing | Failure mode | What produces it | What you see in the log store | |---|---|---| | **No active request context** | process startup, scheduled jobs, background compaction, retry workers, work that continues after the response was written | lines with the field absent or empty, often the ones explaining the failure | | **Context lost on an asynchronous hop** | submitting to a thread pool, a callback, publishing to and consuming from a queue, a reactive or coroutine boundary | the first lines carry the ID and the rest do not; a consumer's lines belong to a different ID or to none | | **Logger never wired to the context** | a third-party library logging through its own channel, a proxy or sidecar's access log, the runtime's own error stream | whole categories of lines that never carry the field at all | The first is structural: some work genuinely has no originating request, and the honest fix is to give background work its own identifier rather than to fake a request ID. The second is the classic bug, because the handoff usually still works — only the context is lost, so nothing fails loudly. The third is an integration gap that shows up as a blind spot in exactly the layer that saw the network error. ## Why partial stamping is worse than it looks A corpus where most lines carry the ID and some do not produces **silent partial recall**. You filter on the ID, get twenty-seven lines, see a four-second gap between two of them, and conclude the request was blocked. In reality the work in that gap ran on a pool thread that lost the context and wrote forty-three lines you never saw. Absence of evidence reads as evidence of absence, and it reads that way in the middle of an incident when nobody is inclined to question the search. That is why the useful operational habit is to measure completeness rather than assume it: - Track the **proportion of log records with an empty or absent trace field**, per service, as an ordinary metric, and treat a rise in it as a regression. - After adding an asynchronous path, pick a recent request, filter by its ID, and check that lines appear from every stage you expect. - Accept and document the entries that legitimately cannot carry one — anything written before the request is identified, such as connection setup or routing failures. The payoff for getting this right is not just faster grepping. A complete, ID-stamped log corpus is the hop that every other correlation workflow lands on: whatever gets you to a specific request, logs are usually where you finish reading.