In Jaeger, what path does a span take from an instrumented process into storage, and where can spans be dropped along it?
answer
- Spans do not go straight to storage
- Every hop has a bounded buffer
- One tier speaks UDP and forgets
- Kafka buys durability the queue cannot
- The read path is a separate service
basics
~20 sA span leaves the instrumented process through a bounded in-process reporter queue, reaches a Jaeger collector, and is written to storage, optionally via Kafka and an ingester. Every hop buffers, and every buffer drops spans when it fills.
solid answer
~50 sInstrumented code hands a finished span to an in-process reporter, which batches it on a **bounded queue** so the request thread never blocks. In Jaeger v1 the batch went over UDP to a local `jaeger-agent`, which forwarded it on; current deployments export straight to `jaeger-collector`, which validates spans and writes them to storage through another bounded in-memory queue — or publishes them to **Kafka**, where a `jaeger-ingester` consumes and writes them. `jaeger-query` reads spans back for the UI and is never in the write path. Loss is designed in at three points: the reporter queue drops when the app produces faster than it exports, the UDP hop is fire-and-forget and silently loses oversized or excess datagrams, and the collector's queue sheds as soon as storage falls behind. Spans are stored individually and assembled into a trace at read time, so loss produces a hole rather than nothing.
code
text · 10 linesapp SDK --[bounded reporter queue]--> agent (v1, UDP) --> jaeger-collector
|
+-----------------------------------------+
| |
[bounded in-memory queue] Kafka topic
| |
storage <---------------------------- jaeger-ingester
^
|
jaeger-query (UI, read path only)go deeper
Be ready to recall the pieces by name: the application exports spans, a collector receives them, a store holds them, and a query service serves the UI. Knowing that the application never writes to the store itself is the point.
Explain the mechanics: which hop batches, which hop is fire-and-forget UDP, and why every queue on the path is bounded. An interviewer expects you to say that spans are stored individually and joined into a trace only at read time.
Show you have debugged this. Walk the pipeline in order, name the counter you would read at each hop, and recognise the signature of a collector shedding spans under storage back-pressure rather than reaching for more replicas.
Own the tradeoff between an in-memory buffer and a durable one. Be able to argue when Kafka's extra stateful system is worth the operational weight, and how you keep one high-volume team from sizing the ingest tier for everybody.
A span in Jaeger is not written where it is created. It travels through a short, explicitly buffered pipeline, and every buffer on that pipeline is **bounded**. That single fact explains most of the "our traces are incomplete" reports that get filed as instrumentation bugs. ## The path, hop by hop 1. **The instrumented process.** Application code — today usually the OpenTelemetry SDK, historically a Jaeger client library — finishes a span and hands it to an in-process reporter. The reporter never blocks the request thread: it puts the span on a bounded queue and a background worker batches and ships it. 2. **The local agent tier (Jaeger v1).** `jaeger-agent` ran as a sidecar or host daemon. It accepted spans pushed over **UDP** (port `6831` for the Thrift compact encoding), batched them, and forwarded them onward to a collector. It also answered clients asking for sampling configuration, over HTTP on port `5778`. 3. **`jaeger-collector`.** Accepts spans over OTLP (`4317` for gRPC, `4318` for HTTP) as well as Jaeger's own protocols, validates them, applies whatever processing is configured, and hands them to a writer. Collectors are stateless and horizontally scaled — nothing requires two spans of the same trace to land on the same one. 4. **Storage, directly or through Kafka.** The collector either writes to the backing store through a bounded in-memory queue, or publishes spans onto a **Kafka** topic, in which case a separate `jaeger-ingester` consumes that topic and performs the writes. 5. **`jaeger-query`.** The read path is a different process entirely. It reads spans back out of storage and serves the UI and API on port `16686`. It is never in the write path, so an expensive search cannot cost you ingest. The property that ties all of this together: **spans are stored individually and a trace is assembled at read time.** No stage of the pipeline holds a complete trace waiting to be committed. That is why a trace renders eighty per cent complete rather than not at all, and why one service failing to export produces a hole in the middle of a trace rather than losing the whole thing. ## Where spans get dropped | Hop | What buffers | What happens when it is full | |---|---|---| | Application reporter | Bounded in-memory queue | Spans dropped; a dropped-span counter increments | | Client to agent, v1 | Nothing — UDP is fire-and-forget | Datagrams silently lost; oversized batches never arrive | | Agent to collector | Bounded queue | Spans dropped, visible in the agent's own metrics | | Collector to storage | Bounded in-memory queue | Spans shed as soon as storage writes fall behind | | Kafka to ingester | Kafka's own topic retention | Nothing dropped until retention expires; lag grows instead | Two of those deserve emphasis. The **UDP hop** was a deliberate choice: it decouples the application from the tracing backend so thoroughly that a collector outage cannot slow a single request. The price is that a burst, an undersized receive buffer on the host, or a batch larger than the datagram limit loses spans with no error raised anywhere. The **collector's queue** is the one that bites in production, because it converts a storage slowdown into span loss within seconds. Its counters are the first place to look, and the tell is dropped spans climbing while collector CPU sits near idle — the collector is not busy, it is waiting on writes. ## Why you would put Kafka in the middle Kafka is the answer to "the collector's memory is the only thing standing between a storage hiccup and lost traces". Publishing to a topic turns the buffer from seconds of RAM into hours of disk, and it lets you scale writers independently from receivers. In a seed-catalogue ordering estate where one team generates seventy per cent of the span volume, it also gives you somewhere to absorb that team's deploy-time bursts without sizing every collector for their peak. What it costs is a second stateful system to operate, consumer lag as a new thing to alert on, and extra end-to-end delay before a trace becomes searchable. ## Diagnosing a hole in a trace Work the pipeline in order rather than guessing: - **Was the span ever created?** A sampling decision taken at trace start is not a drop — the span never existed. Check that first; it is by far the most common false alarm. - **Did it leave the process?** Look at the exporting SDK's own dropped-span and export-failure counters before blaming the backend. - **Did it survive the network hop?** On a v1 agent deployment, check host-level UDP receive errors as well as the agent's metrics; nothing else will tell you. - **Did the collector shed it?** Compare spans received against spans saved, and watch queue depth over the same window. - **Is it late rather than lost?** With Kafka in the path, consumer lag shows up as traces that appear minutes after the request completed.
- What changes operationally once you put Kafka between the collector and storage?The buffer stops being seconds of collector RAM and becomes hours of disk, so a storage outage delays writes instead of dropping spans, and receivers scale independently of writers. In exchange you operate a second stateful system, consumer lag becomes something you must alert on, and traces take longer to become searchable after the request completes.
- Your collector's dropped-span counter is climbing while its CPU sits near idle. Where do you look?At storage write latency. An idle collector with a filling queue is not compute-bound; it is blocked waiting on the backend, and the queue is shedding under back-pressure. Check write latency and error rates on the store, then queue depth over the same window. Adding collector replicas will not help — they all write to the same backend.
- Why was UDP a defensible choice for the client-to-agent hop, and what did it cost?It decouples the application from the tracing backend completely: a collector outage or a slow agent can never add latency to a request or block a thread. The cost is silent loss. A traffic burst, an undersized host receive buffer, or a batch bigger than the datagram limit discards spans with no error surfaced anywhere in the application.
It is a bucket brigade with fixed-size buckets: nobody waits for the person ahead of them, so when the far end slows down the overflow simply goes on the floor, quietly.
saying these in an interview costs you the question
- Says the application writes spans to storage synchronously
- Assumes no span is ever dropped once instrumentation is on
- Describes the Jaeger agent as scraping services for spans
- Thinks a complete trace is assembled before being stored
- Believes adding Kafka removes the need for client-side buffering
- Confuses a sampling decision with a span being dropped