A payments webhook function enqueues work for a second function, but traces stop at the queue; how do you carry trace context across that asynchronous hop?
answer
- a queue has no request headers
- the poller starts a new root
- inject on send, extract on receive
- one batch, many parents
- span links
basics
~20 sTrace context crosses HTTP calls in headers, but a queued message only carries what the producer puts in it. Inject the context into message attributes on send, extract it per message in the consumer, and use per-message spans or links for batches.
solid answer
~50 sTracing libraries continue a trace over HTTP by writing the trace context into request headers. A queue is not an HTTP call: the producer's tracer adds nothing to a message unless the queue client is instrumented, and the consumer is invoked by the platform's poller, so its spans start a new trace unless the code supplies the producer's context. The fix is to inject the context, in the standard `traceparent` format, into each message's attributes on send, and to extract it per message in the consumer before starting spans. A consumer invocation often receives a batch from many traces, and a span has at most one parent, so create one span per message parented to that message's context, or put span links on the batch span. Flush spans before the handler returns, and log the event id and trace id on both sides so the hop can still be joined when tracing fails.
code
pseudocode · 19 lines# producer function
attributes = {}
propagator.inject(tracer.currentContext(), into = attributes) # writes traceparent
queue.send(body = payment, attributes = attributes)
# consumer function, invoked by the platform with a batch
contexts = [propagator.extract(from = m.attributes) for m in event.messages]
batchSpan = tracer.startSpan("receive-batch", links = contexts)
try:
for message, ctx in zip(event.messages, contexts):
span = tracer.startSpan("process-payment", parent = ctx,
attributes = {eventId: message.id})
try:
process(message)
finally:
span.end()
finally:
batchSpan.end()
tracer.flush() # export before returning, while the sandbox still runsgo deeper
Recall that a queue does not carry HTTP headers, so the sending function has to put the trace context into the message itself.
Explain inject on send and extract on receive, and why the context belongs in message attributes rather than in the payload body.
Show how you handle a batch from many traces with per-message spans or links, flush before returning, and keep logs keyed by event id as the fallback join.
Discuss the propagation format and the log fields a platform team mandates for every function, so a trace survives any queue, topic or tracing vendor.
## Why the trace stops at the queue Tracing libraries continue a trace across an HTTP call by writing the **trace context** (the trace id, the parent span id and a sampled flag, usually in the standard `traceparent` header) into the outgoing request and reading it on the other side. A queue between two functions is not an HTTP call: - The producer's tracer does not add anything to a message unless the queue client is instrumented to do so. - The consumer is not called by the producer. The platform's poller reads the queue and invokes the consumer, often with a **batch** of messages from many different producers. - The platform starts an invocation span for that call, and it has no parent unless your code supplies one. The result is two traces: one ending at "send message" and one starting at "receive batch", with nothing joining them. ## Carrying the context: inject and extract 1. **Inject on send.** In the producer, ask the tracing library's propagator to write the current context into the message's **attributes** or metadata, not into the payload body that other consumers parse. 2. **Extract on receive.** In the consumer, read the context from each message before starting any span for it. 3. **Start one span per message** with that context as its parent, and record the event id on it. 4. **End and export** every span before the handler returns. Attributes are the right home for the context for three reasons. They travel with the message without changing the payload schema that other consumers depend on. Queue instrumentation in tracing libraries knows how to read them, so an instrumented consumer may extract the context with little or no code. And they keep tracing metadata out of data that is archived, audited or replayed as business records. Some platforms' tracing services do this automatically for their own queue integrations. That only holds while every hop uses the same service; once you mix in your own tracer or a queue from elsewhere, agree on one header format and inject it yourself. ## The batch problem A span has at most one parent. A consumer invocation that receives messages from five different traces cannot belong to all five. Two shapes work: - **Per-message spans**, each parented to its own message's context, so every upstream trace continues through its own message. - **Span links**: the batch's invocation span carries a link to every message's context, so a trace viewer can jump from the batch to each producer and back. Making the first message's context the parent of the whole batch is the common mistake: the other four traces then appear to end at the queue, and the one that "continues" shows work done for other requests. ## What still breaks in production - **Telemetry lost at the end of the invocation.** Spans buffered in memory may never be exported if the sandbox is paused right after the handler returns, so flush before returning. - **Sampling.** Carry the sampled flag with the context, so the consumer keeps or drops the same traces as the producer did. - **Fan-out through a topic.** Each subscriber's delivery must preserve message attributes; some delivery modes drop them unless configured to keep them. - **Redrives.** A message replayed after a failure keeps its original context, so its spans can appear long after the trace began. That is correct, not corruption. ## When traces fail anyway: structured logs Write **structured logs** on both sides with the same fields: the event id, the trace id, the function version and the platform's request id. Even with a broken trace, a log query on the event id joins the producer's line to the consumer's, and the version field tells you which release handled each side. ## The hops compared | Hop | Who carries the context | What you must do | |---|---|---| | Function to function over HTTP | An instrumented HTTP client and server | Usually nothing beyond instrumentation | | Function to queue to function | Your code, via message attributes | Inject on send, extract per message | | A batch from many producers | Your code | Per-message spans, or links on the batch span |
- Why might the consumer's spans be missing entirely rather than just disconnected?Many tracers buffer spans and export them in the background. If the handler returns before the buffer is flushed, the sandbox can be paused or recycled with the spans still in memory. Flush explicitly before returning, on the error path as well as the success path.
- What should both functions log so the hop can be joined when tracing fails?The same structured fields on both sides: the event id or idempotency key, the trace id, the executed function version and the platform's request id. A query on the event id then joins the producer's line to the consumer's, and the version shows which release handled each side.
- The platform's tracing joins producer and consumer automatically for any queue.
- Trace context only matters for synchronous HTTP calls.
- A batch invocation can take the first message's trace as its parent.
- Putting the trace id in the message body is enough for the consumer's tracer.
- If the trace breaks, logs cannot be joined across the hop either.