How does trace context propagate across non-HTTP boundaries like Kafka or RabbitMQ, and what do you do when auto-instrumentation isn't available?

seniorshould knowfreq 40%

answer

Carrier = message headers instead of HTTP headers
Propagator.inject on send, extract on receive
Kafka: observation-enabled -> traceparent in RecordHeaders
no thread-local across the broker — context lives in headers
batch listeners: many contexts, decide per-record vs batch-root

basics

~20 s

Context travels as message headers instead of HTTP headers. Spring's Kafka/RabbitMQ instrumentation injects traceparent/baggage into producer record headers and extracts them on the consumer. Where no auto-instrumentation exists, use the Propagator API to inject on send and extract on receive manually.

solid answer

~40 s

Propagation isn't HTTP-specific — the trace context just needs a carrier with string key/values. For messaging, that carrier is the message headers (Kafka RecordHeaders, AMQP message properties). Spring's Kafka and RabbitMQ observability integrations inject the current context (traceparent + declared baggage) into the producer's headers and extract it on the consumer, so the consumer span becomes a child of the producer span across the broker. For transports without built-in support, you drive Micrometer's Propagator directly: on send, propagator.inject(currentContext, carrier, (c, key, val) -> c.put(key, val)); on receive, propagator.extract(carrier, (c, key) -> c.get(key)) to rebuild the context, then open a span in that scope. Key gotchas: async handoff means you must not rely on thread-locals surviving the broker; and one producer batching many messages needs per-message context, not one shared span.

Propagation isn't HTTP-specific — the trace context just needs a **carrier** with string key/values. For messaging, that carrier is the **message headers** (Kafka `RecordHeaders`, AMQP message properties). Spring's Kafka and RabbitMQ observability integrations inject the current context (traceparent + declared baggage) into the producer's headers and extract it on the consumer, so the consumer span becomes a child of the producer span across the broker. For transports without built-in support, you drive **Micrometer's `Propagator`** directly: on send, `propagator.inject(currentContext, carrier, (c, key, val) -> c.put(key, val))`; on receive, `propagator.extract(carrier, (c, key) -> c.get(key))` to rebuild the context, then open a span in that scope. Key gotchas: async handoff means you must not rely on thread-locals surviving the broker; and one producer batching many messages needs per-message context, not one shared span.

code

java · 29 lines

java

// Manual propagation over a transport WITHOUT auto-instrumentation.
@Component
class TracedPublisher {
    private final Propagator propagator;
    private final Tracer tracer;
    private final CustomBroker broker;

    TracedPublisher(Propagator propagator, Tracer tracer, CustomBroker broker) {
        this.propagator = propagator; this.tracer = tracer; this.broker = broker;
    }

    void publish(Message msg) {
        // Inject current trace context + baggage into the message's headers (the carrier).
        propagator.inject(tracer.currentTraceContext().context(), msg.headers(),
                (headers, key, value) -> headers.put(key, value));
        broker.send(msg);
    }

    // Consumer side rebuilds the span from the carrier:
    void onMessage(Message msg) {
        Span span = propagator.extract(msg.headers(), (headers, key) -> headers.get(key))
                              .name("custom.consume").start();
        try (Tracer.SpanInScope ws = tracer.withSpan(span)) {
            handle(msg);
        } finally {
            span.end();
        }
    }
}

go deeper

for a junior

Know context can travel on message headers, not only HTTP headers.

for a middle

Explain that Spring Kafka/AMQP observation injects/extracts context and enables it on both sides.

for a senior

Use the Propagator inject/extract primitive for uninstrumented transports and handle batch/async scope correctly.

for a principal

Design consistent messaging-trace policy: per-record vs batch tracing, baggage cost on high-volume topics, and replayed-message semantics.

Propagation is transport-agnostic. The trace context and baggage are just string key/value pairs; anything that can carry string metadata can be a carrier. For HTTP the carrier is request headers; for messaging it's the message headers/properties. The tracer's Propagator abstraction has two operations: inject (write context into a carrier via a setter) and extract (read context from a carrier via a getter). HTTP instrumentation is just a built-in inject/extract over HTTP headers.

Kafka. With Spring Kafka observability enabled (spring.kafka.template.observation-enabled=true and observation on the listener container factory), the producer instrumentation injects traceparent (+ any remote-field baggage) into the ProducerRecord's RecordHeaders. On the consumer side, the listener instrumentation extracts those headers and starts a consumer span whose parent is the producer span — so the trace spans the broker even though producer and consumer are different processes on different threads. Note the relationship is typically parent→child across the async gap; the producer span usually finishes before the consumer runs, so the consumer span links back rather than nesting live.

RabbitMQ / AMQP. Similarly, Spring AMQP writes the context into message properties/headers on publish and reads it on delivery when observation is enabled on the RabbitTemplate and listener containers.

When there's no auto-instrumentation (a custom transport, a raw SQS/SNS client, a file-based handoff): drive the Propagator yourself.

Producer/inject:

java

propagator.inject(currentTraceContext.context(), record.headers(),
    (headers, key, value) -> headers.add(key, value.getBytes(UTF_8)));

Consumer/extract: rebuild a Span from the carrier, then run in its scope:

java

Span span = propagator.extract(record.headers(), (headers, key) -> ...)
                      .name("consume").start();
try (var ws = tracer.withSpan(span)) { handle(record); }
finally { span.end(); }

(In practice you'd prefer Micrometer's higher-level Observation API, but the Propagator inject/extract is the primitive.)

Gotchas.

Thread-locals don't cross the broker. The context is only recoverable from the message headers on the consumer side; there is no shared thread-local. If headers aren't injected, the consumer starts a brand-new trace — a very common cause of 'my Kafka consumers show disconnected traces'.
Batching / fan-out. A producer sending many records in a loop must inject the current per-message context. If you open one span around a batch, every message links to the same producer span (often fine), but if each message logically belongs to a different inbound trace you must scope each send separately.
Consumer batch listeners. A @KafkaListener consuming a batch receives many records with potentially different trace contexts; a single listener invocation can't be one child span of all of them. Decide whether to trace per-record or treat the batch as its own root.
Baggage over messaging. Baggage remote-fields propagate the same way (extra headers), so header-size and PII concerns are amplified for high-volume topics.
Header key casing / byte encoding. Kafka headers are byte[]; encode/decode consistently (UTF-8) or extraction silently fails.
Poison/replayed messages. Old messages carry old (possibly long-finished) trace IDs; that's expected — the consumer span links to a historical trace.

When to use manual propagation. Only when a transport lacks Spring/Micrometer instrumentation. Prefer enabling the built-in observation support first; hand-rolled inject/extract is error-prone and easy to get subtly wrong (missing baggage, wrong scope lifetime).

**Propagation is transport-agnostic.** The trace context and baggage are just string key/value pairs; anything that can carry string metadata can be a **carrier**. For HTTP the carrier is request headers; for messaging it's the **message headers/properties**. The tracer's **`Propagator`** abstraction has two operations: **inject** (write context into a carrier via a *setter*) and **extract** (read context from a carrier via a *getter*). HTTP instrumentation is just a built-in inject/extract over HTTP headers. **Kafka.** With Spring Kafka observability enabled (`spring.kafka.template.observation-enabled=true` and observation on the listener container factory), the producer instrumentation **injects** `traceparent` (+ any remote-field baggage) into the `ProducerRecord`'s `RecordHeaders`. On the consumer side, the listener instrumentation **extracts** those headers and starts a **consumer span** whose parent is the producer span — so the trace spans the broker even though producer and consumer are different processes on different threads. Note the relationship is typically parent→child *across* the async gap; the producer span usually finishes before the consumer runs, so the consumer span links back rather than nesting live. **RabbitMQ / AMQP.** Similarly, Spring AMQP writes the context into message properties/headers on publish and reads it on delivery when observation is enabled on the `RabbitTemplate` and listener containers. **When there's no auto-instrumentation** (a custom transport, a raw SQS/SNS client, a file-based handoff): drive the `Propagator` yourself. - **Producer/inject:** ```java propagator.inject(currentTraceContext.context(), record.headers(), (headers, key, value) -> headers.add(key, value.getBytes(UTF_8))); ``` - **Consumer/extract:** rebuild a `Span` from the carrier, then run in its scope: ```java Span span = propagator.extract(record.headers(), (headers, key) -> ...) .name("consume").start(); try (var ws = tracer.withSpan(span)) { handle(record); } finally { span.end(); } ``` (In practice you'd prefer Micrometer's higher-level `Observation` API, but the `Propagator` inject/extract is the primitive.) **Gotchas.** - **Thread-locals don't cross the broker.** The context is only recoverable from the message headers on the consumer side; there is no shared thread-local. If headers aren't injected, the consumer starts a brand-new trace — a very common cause of 'my Kafka consumers show disconnected traces'. - **Batching / fan-out.** A producer sending many records in a loop must inject the *current per-message* context. If you open one span around a batch, every message links to the same producer span (often fine), but if each message logically belongs to a different inbound trace you must scope each send separately. - **Consumer batch listeners.** A `@KafkaListener` consuming a batch receives many records with potentially different trace contexts; a single listener invocation can't be one child span of all of them. Decide whether to trace per-record or treat the batch as its own root. - **Baggage over messaging.** Baggage remote-fields propagate the same way (extra headers), so header-size and PII concerns are amplified for high-volume topics. - **Header key casing / byte encoding.** Kafka headers are `byte[]`; encode/decode consistently (UTF-8) or extraction silently fails. - **Poison/replayed messages.** Old messages carry old (possibly long-finished) trace IDs; that's expected — the consumer span links to a historical trace. **When to use manual propagation.** Only when a transport lacks Spring/Micrometer instrumentation. Prefer enabling the built-in observation support first; hand-rolled inject/extract is error-prone and easy to get subtly wrong (missing baggage, wrong scope lifetime).

Your Kafka consumers each show up as separate root traces instead of continuing the producer's trace. What's wrong?
The context isn't being injected into/extracted from record headers — usually observation isn't enabled (producer `observation-enabled`, or the listener container factory lacks an ObservationRegistry). Without headers there's no thread-local to fall back on across the broker, so the consumer starts fresh. Enable observation on both sides.
How do producer and consumer spans relate across a broker — parent/child nesting or something else?
It's a cross-process parent→child link, not live nesting. The producer span typically ends before the message is consumed, so the consumer span references the producer span as its parent (a link/follows-from relationship) rather than being an active child within the same live scope.

saying these in an interview costs you the question

Assuming thread-locals carry context across a message broker
Believing HTTP is the only thing tracing can propagate over
Hand-rolling inject/extract when Spring's messaging observation would work
Ignoring that a batch listener may hold records from many different traces
Forgetting Kafka headers are byte[] and need consistent encoding

How does trace context propagate across non-HTTP boundaries like Kafka or RabbitMQ, and what do you do when auto-instrumentation isn't available?

§ answer

basics

solid answer

code

§ go deeper

▾level notes4 levels

▾deep dive~3 min

▾common follow-ups2 questions

▾red flags5 flags

§ context

▾related questions5 questions

▾related topics4 topics

answer

go deeper

context