Walk through how a distributed tracing system such as Jaeger or Zipkin, instrumented via OpenTelemetry, tracks a single request as it flows through five microservices: what gets generated, what gets passed between services on the wire, and how does the backend reassemble it into one picture?

## The unit of data Distributed tracing works by treating a single logical request as a tree of timed operations called **spans**, and by propagating enough context on every network call for each service to correctly place its own spans into that tree. The unit of data is the span: a record that has - a name (e.g. 'GET /cart' or 'db.query'), - a start timestamp, - a duration, - a status (ok or error), - a set of key-value attributes describing what happened, - and two identifiers, its own span ID and the trace ID of the request it belongs to. A **trace** is simply the full set of spans that share one trace ID, and the parent-child pointers between those spans, each span except the very first records a parent span ID, describe the causal structure: which operation triggered which other operation, and in what order. ## Hop by hop Concretely, the mechanism unfolds hop by hop. 1. The first service to touch the request, say an API gateway, has no incoming trace context, so its tracing library, OpenTelemetry's SDK in most modern stacks, mints a fresh trace ID and opens a **root span**. 2. If it needs to call an inventory service to serve the request, it injects the current trace context into the outgoing HTTP call, in practice as a W3C `traceparent` header of the shape `00-<trace-id>-<span-id>-<flags>`, where the span-id in that header is the gateway's current span, and the flags include a sampled bit. 3. The inventory service extracts that header on receipt, sees it belongs to an existing trace, and opens its own span as a child of the span ID it just read, with the same trace ID. 4. If the inventory service in turn calls a database or a pricing service, it repeats the same injection step with its own span now as the parent. 5. Each span, once finished, is exported, usually asynchronously and batched, over gRPC or HTTP, to a collector process, which forwards it to a storage backend, indexed by trace ID. A UI like the Jaeger or Zipkin web console then queries all spans for one trace ID and renders them as a waterfall or flame graph, ordered by start time and nested by parent-child relationship, which is what lets an engineer see at a glance that the pricing service's database call consumed 380ms out of a 420ms total request. ## Why grouping by log line is not enough The reason this exists, rather than just using logs and correlation IDs, is that logs alone give you grouping but not causal shape or precise timing. A trace answers exactly where the two seconds went, and in what order the five services called each other, in a way that scanning timestamped log lines across five services cannot reliably reconstruct, especially once retries, parallel fan-out calls, and asynchronous work are involved. Tracing also gives you span attributes as structured, queryable metadata (HTTP status code, DB statement, customer tier) that lets you slice traces, for example show me all traces where the payments span had status equal to error, far more cheaply than grepping unstructured logs. ## What it costs The trade-offs are real. - Every service needs instrumentation, either manual, wrapping code in spans, or automatic, agents that instrument common libraries like HTTP clients and JDBC drivers. - Every network hop, including message queues and especially any client the team doesn't fully control, must correctly propagate the trace-context header or the trace breaks into disconnected fragments. - Storing a full span tree for every single request at high request-per-second volumes is expensive in both storage and the write throughput the collector and backend must sustain, which is why almost no production system traces one hundred percent of traffic, this is the sampling problem, a distinct but closely related concern. - There is also a real engineering cost to running the collector pipeline itself and to keeping instrumentation libraries current as dependencies evolve. ## Failure modes Failure modes show up in a few recognizable shapes. - **A broken trace** happens when some hop, commonly a message queue consumer, a serverless function cold-starting, or a third-party API, fails to propagate the `traceparent` header, so the trace silently splits into two unconnected trace IDs and the UI shows an incomplete picture with a suspicious gap. - **A clock skew issue** happens when spans from different hosts have drifted system clocks, making the waterfall view show a child span apparently starting before its parent. - **Over-instrumentation**, spans for every trivial function call, produces noisy traces that are hard to read and expensive to store, while **under-instrumentation** leaves exactly the slow, opaque hop uninstrumented, which tends to be the one you needed visibility into during an incident. ## The lineage A concrete, well-known lineage: - Google's internal **Dapper** paper described exactly this span, trace-ID, parent-ID model at Google scale; - Twitter's **Zipkin** and Uber's **Jaeger** are open-source systems built on the same ideas; - and **OpenTelemetry** is the current CNCF-backed standard that unifies instrumentation APIs and the wire format, W3C Trace Context, so that a span emitted by a Java service and a span emitted by a Go service can be correctly stitched into one trace by any compliant backend.