skip to content

How is the same traceId carried across service boundaries so logs from different services correlate, and how do those logs tie back to a trace?

level: seniorimportance: should knowfreq 48%

answer

  1. Propagator inject/extract
  2. W3C traceparent: 00-traceId-parentSpanId-flags
  3. Brave = B3 headers
  4. RestClient/WebClient inject, server filter extracts
  5. same traceId in logs AND tracing backend = pivot

basics

~20 s

The traceId is sent to downstream services in an HTTP header (by default the W3C traceparent header). The downstream service extracts it, continues the same trace, and stamps the same traceId into its own logs. Because logs and the tracing backend share that traceId, you can jump between them.

solid answer

~50 s

Micrometer Tracing uses a **Propagator** to inject the current trace context into outgoing requests and extract it from incoming ones. By default (OTel bridge) it uses the **W3C Trace Context** `traceparent` header, which encodes traceId, the caller's spanId (as parent), and sampling flags; Brave can also use B3 headers. Instrumented clients — `RestTemplate`, `RestClient`, `WebClient`, and the server-side filter — do this automatically, so service B extracts the traceId from the header, starts a child span under the same trace, and stamps that same traceId into its MDC and logs. The result: services A and B print the identical traceId. Because your tracing backend (Zipkin/Tempo/Jaeger) indexes spans by that traceId, and your logs contain it, you can pivot from a log line to the full distributed trace and back — that's 'tying logs to traces'.

code

java · 19 lines
java
// Auto-instrumented clients propagate the header for you:
@Bean RestClient restClient(RestClient.Builder builder) {
  return builder.baseUrl("http://downstream").build();
}

// For a NON-instrumented / raw client you inject manually via the Propagator:
@Component
class ManualCaller {
  private final Tracer tracer;
  private final Propagator propagator;
  ManualCaller(Tracer tracer, Propagator propagator) {
    this.tracer = tracer; this.propagator = propagator;
  }
  void call(HttpRequestBuilder req) {
    // writes traceparent (or B3) so the callee continues the same trace
    propagator.inject(tracer.currentTraceContext().context(),
        req, (carrier, key, value) -> carrier.header(key, value));
  }
}

go deeper

for a junior

Knows the ID somehow travels between services in a header.

for a middle

Names traceparent/B3 and that instrumented clients propagate automatically.

for a senior

Explains inject/extract, format config, and how logs pivot to traces via shared traceId.

for a principal

Owns fleet-wide propagation format standardization, gateway header allowlists, non-HTTP hops, and log/trace/metric (exemplar) linkage strategy.

## The mechanism: propagation headers A distributed trace only works if the traceId travels between processes. Micrometer Tracing does this through a **Propagator** abstraction with two operations: - **inject** — write the current trace context into an outgoing carrier (HTTP headers). - **extract** — read it from an incoming carrier and continue the trace. ### Default format: W3C Trace Context With the OpenTelemetry bridge, the default is the **W3C `traceparent`** header: ``` traceparent: 00-<32-hex-traceId>-<16-hex-parentSpanId>-<flags> ``` - `00` = version - traceId = the shared correlation ID for the whole request - parentSpanId = the caller's span, so the callee nests correctly - flags = sampling decision (e.g., `01` = sampled) Optionally a `tracestate` header carries vendor data. The **Brave** bridge historically uses **B3** headers (`X-B3-TraceId`, `X-B3-SpanId`, `X-B3-Sampled`, or a single `b3` header). You can configure which format via `management.tracing.propagation.type` (`W3C` or `B3`). ## Who injects/extracts automatically Spring Boot instruments common integration points so you rarely touch the Propagator directly: - **Outbound**: `RestTemplate` (via an interceptor), `RestClient`, `WebClient` (via a filter/exchange filter), and messaging clients — inject `traceparent`. - **Inbound**: a Servlet/WebFlux **Observation** filter extracts the header, creates a server span whose traceId matches the caller, and scopes it so MDC is populated for that request's logs. So across the call chain A → B → C, all three log the **same traceId** (each with its own spanId), and the spans form a parent/child tree. ## Tying logs to traces This is the payoff of correlation: 1. Your **logs** contain `traceId` (via the correlation pattern / MDC). 2. Your **tracing backend** (Zipkin, Grafana Tempo, Jaeger) stores each span indexed by that **same traceId**. 3. Therefore, from a suspicious log line you copy the traceId and open the full trace waterfall; conversely, from a slow span you search logs by its traceId to read the detailed messages. Grafana's Loki↔Tempo, or an APM's log-trace linking, automate this pivot. 4. Metrics can join in too via **exemplars** — sampled metric data points carrying a traceId — letting you jump from a latency spike on a histogram to an example trace. ## Gotchas - **A proxy or gateway that strips unknown headers** breaks propagation — the callee starts a brand-new trace, and logs no longer correlate across the boundary. Allowlist `traceparent`/`tracestate` (or B3) headers. - **Format mismatch**: if service A emits B3 but service B only reads W3C (or vice versa), extraction fails silently and a new trace begins. Standardize the propagation type across the fleet. - **Manual HTTP clients** (raw `HttpClient`, some SDKs) aren't instrumented; you must inject headers yourself via the `Propagator`, or the trace breaks at that hop. - **Sampling consistency**: the sampling flag in `traceparent` is honored downstream, so an unsampled trace stays unsampled end-to-end — but IDs still stamp into logs regardless of sampling, so log correlation survives even when the trace isn't exported. - **Non-HTTP hops** (Kafka, RabbitMQ) need messaging instrumentation to carry context in message headers. ## When to use Always, in any multi-service system where you want a single request's logs and trace stitched together. The cost is a couple of headers; the benefit is being able to follow one request across the entire system.

  • An API gateway sits in front of service B; suddenly B's logs show a different traceId than A's. What's the most likely cause?
    The gateway (or a proxy) is stripping the traceparent/B3 header, so B can't extract the incoming context and starts a fresh trace. Allowlist the propagation headers through the gateway.
  • What's the difference between the correlation you get in logs and an exported trace, with respect to sampling?
    IDs are stamped into MDC/logs regardless of the sampling decision, so log correlation always works. Sampling only controls whether spans are exported to the tracing backend, so an unsampled request still has correlated logs but no visible trace waterfall.

saying these in an interview costs you the question

  • Thinking the traceId travels in the request body or a cookie rather than a propagation header
  • Assuming every HTTP client auto-propagates, including raw java.net.http clients
  • Not realizing header-stripping proxies silently break cross-service correlation
  • Mixing B3 and W3C formats across services and expecting correlation to work

context

resolving links…