skip to content

Jaeger

The open-source trace backend: it collects spans, applies sampling, stores traces in Cassandra or Elasticsearch, and gives you the waterfall view plus a service dependency graph. Interviews ask how you use a trace to attribute latency to one hop instead of guessing.

questions

4

In Jaeger, what path does a span take from an instrumented process into storage, and where can spans be dropped along it?

level: middleimportance: must knowfreq 60%

answer

  1. Spans do not go straight to storage
  2. Every hop has a bounded buffer
  3. One tier speaks UDP and forgets
  4. Kafka buys durability the queue cannot
  5. The read path is a separate service

basics

~20 s

A span leaves the instrumented process through a bounded in-process reporter queue, reaches a Jaeger collector, and is written to storage, optionally via Kafka and an ingester. Every hop buffers, and every buffer drops spans when it fills.

solid answer

~50 s

Instrumented code hands a finished span to an in-process reporter, which batches it on a **bounded queue** so the request thread never blocks. In Jaeger v1 the batch went over UDP to a local `jaeger-agent`, which forwarded it on; current deployments export straight to `jaeger-collector`, which validates spans and writes them to storage through another bounded in-memory queue — or publishes them to **Kafka**, where a `jaeger-ingester` consumes and writes them. `jaeger-query` reads spans back for the UI and is never in the write path. Loss is designed in at three points: the reporter queue drops when the app produces faster than it exports, the UDP hop is fire-and-forget and silently loses oversized or excess datagrams, and the collector's queue sheds as soon as storage falls behind. Spans are stored individually and assembled into a trace at read time, so loss produces a hole rather than nothing.

code

text · 10 lines
text
app SDK --[bounded reporter queue]--> agent (v1, UDP) --> jaeger-collector
                                                              |
                    +-----------------------------------------+
                    |                                         |
         [bounded in-memory queue]                      Kafka topic
                    |                                         |
                 storage <---------------------------- jaeger-ingester
                    ^
                    |
              jaeger-query (UI, read path only)

go deeper

for a junior

Be ready to recall the pieces by name: the application exports spans, a collector receives them, a store holds them, and a query service serves the UI. Knowing that the application never writes to the store itself is the point.

for a middle

Explain the mechanics: which hop batches, which hop is fire-and-forget UDP, and why every queue on the path is bounded. An interviewer expects you to say that spans are stored individually and joined into a trace only at read time.

for a senior

Show you have debugged this. Walk the pipeline in order, name the counter you would read at each hop, and recognise the signature of a collector shedding spans under storage back-pressure rather than reaching for more replicas.

for a principal

Own the tradeoff between an in-memory buffer and a durable one. Be able to argue when Kafka's extra stateful system is worth the operational weight, and how you keep one high-volume team from sizing the ingest tier for everybody.

A span in Jaeger is not written where it is created. It travels through a short, explicitly buffered pipeline, and every buffer on that pipeline is **bounded**. That single fact explains most of the "our traces are incomplete" reports that get filed as instrumentation bugs. ## The path, hop by hop 1. **The instrumented process.** Application code — today usually the OpenTelemetry SDK, historically a Jaeger client library — finishes a span and hands it to an in-process reporter. The reporter never blocks the request thread: it puts the span on a bounded queue and a background worker batches and ships it. 2. **The local agent tier (Jaeger v1).** `jaeger-agent` ran as a sidecar or host daemon. It accepted spans pushed over **UDP** (port `6831` for the Thrift compact encoding), batched them, and forwarded them onward to a collector. It also answered clients asking for sampling configuration, over HTTP on port `5778`. 3. **`jaeger-collector`.** Accepts spans over OTLP (`4317` for gRPC, `4318` for HTTP) as well as Jaeger's own protocols, validates them, applies whatever processing is configured, and hands them to a writer. Collectors are stateless and horizontally scaled — nothing requires two spans of the same trace to land on the same one. 4. **Storage, directly or through Kafka.** The collector either writes to the backing store through a bounded in-memory queue, or publishes spans onto a **Kafka** topic, in which case a separate `jaeger-ingester` consumes that topic and performs the writes. 5. **`jaeger-query`.** The read path is a different process entirely. It reads spans back out of storage and serves the UI and API on port `16686`. It is never in the write path, so an expensive search cannot cost you ingest. The property that ties all of this together: **spans are stored individually and a trace is assembled at read time.** No stage of the pipeline holds a complete trace waiting to be committed. That is why a trace renders eighty per cent complete rather than not at all, and why one service failing to export produces a hole in the middle of a trace rather than losing the whole thing. ## Where spans get dropped | Hop | What buffers | What happens when it is full | |---|---|---| | Application reporter | Bounded in-memory queue | Spans dropped; a dropped-span counter increments | | Client to agent, v1 | Nothing — UDP is fire-and-forget | Datagrams silently lost; oversized batches never arrive | | Agent to collector | Bounded queue | Spans dropped, visible in the agent's own metrics | | Collector to storage | Bounded in-memory queue | Spans shed as soon as storage writes fall behind | | Kafka to ingester | Kafka's own topic retention | Nothing dropped until retention expires; lag grows instead | Two of those deserve emphasis. The **UDP hop** was a deliberate choice: it decouples the application from the tracing backend so thoroughly that a collector outage cannot slow a single request. The price is that a burst, an undersized receive buffer on the host, or a batch larger than the datagram limit loses spans with no error raised anywhere. The **collector's queue** is the one that bites in production, because it converts a storage slowdown into span loss within seconds. Its counters are the first place to look, and the tell is dropped spans climbing while collector CPU sits near idle — the collector is not busy, it is waiting on writes. ## Why you would put Kafka in the middle Kafka is the answer to "the collector's memory is the only thing standing between a storage hiccup and lost traces". Publishing to a topic turns the buffer from seconds of RAM into hours of disk, and it lets you scale writers independently from receivers. In a seed-catalogue ordering estate where one team generates seventy per cent of the span volume, it also gives you somewhere to absorb that team's deploy-time bursts without sizing every collector for their peak. What it costs is a second stateful system to operate, consumer lag as a new thing to alert on, and extra end-to-end delay before a trace becomes searchable. ## Diagnosing a hole in a trace Work the pipeline in order rather than guessing: - **Was the span ever created?** A sampling decision taken at trace start is not a drop — the span never existed. Check that first; it is by far the most common false alarm. - **Did it leave the process?** Look at the exporting SDK's own dropped-span and export-failure counters before blaming the backend. - **Did it survive the network hop?** On a v1 agent deployment, check host-level UDP receive errors as well as the agent's metrics; nothing else will tell you. - **Did the collector shed it?** Compare spans received against spans saved, and watch queue depth over the same window. - **Is it late rather than lost?** With Kafka in the path, consumer lag shows up as traces that appear minutes after the request completed.

  • What changes operationally once you put Kafka between the collector and storage?
    The buffer stops being seconds of collector RAM and becomes hours of disk, so a storage outage delays writes instead of dropping spans, and receivers scale independently of writers. In exchange you operate a second stateful system, consumer lag becomes something you must alert on, and traces take longer to become searchable after the request completes.
  • Your collector's dropped-span counter is climbing while its CPU sits near idle. Where do you look?
    At storage write latency. An idle collector with a filling queue is not compute-bound; it is blocked waiting on the backend, and the queue is shedding under back-pressure. Check write latency and error rates on the store, then queue depth over the same window. Adding collector replicas will not help — they all write to the same backend.
  • Why was UDP a defensible choice for the client-to-agent hop, and what did it cost?
    It decouples the application from the tracing backend completely: a collector outage or a slow agent can never add latency to a request or block a thread. The cost is silent loss. A traffic burst, an undersized host receive buffer, or a batch bigger than the datagram limit discards spans with no error surfaced anywhere in the application.

It is a bucket brigade with fixed-size buckets: nobody waits for the person ahead of them, so when the far end slows down the overflow simply goes on the floor, quietly.

  • Says the application writes spans to storage synchronously
  • Assumes no span is ever dropped once instrumentation is on
  • Describes the Jaeger agent as scraping services for spans
  • Thinks a complete trace is assembled before being stored
  • Believes adding Kafka removes the need for client-side buffering
  • Confuses a sampling decision with a span being dropped

Answers here assume Jaeger v1's component split — agent, collector, ingester and query. Jaeger v2 is built on the OpenTelemetry Collector framework and drops the separate agent process: applications export OTLP to a collector directly, or through a collector deployed in the local agent role. The queueing and drop points described are unchanged.

open as a page

In Jaeger's trace waterfall, how do you tell a genuinely slow service from one merely waiting on a downstream call?

level: middleimportance: must knowfreq 66%

basics

~20 s

Compare a span's duration with the time its children cover. A parent's bar encloses its children, so the slow hop is the one with large self time — duration minus child coverage — not simply the longest bar on the screen.

open as a page

In Jaeger, what does serving sampling configuration centrally to clients solve, and what can it not solve?

level: seniorimportance: should knowfreq 42%

basics

~20 s

Jaeger clients poll the backend for their sampling rate, so per-service and per-operation rates change without a redeploy. Central configuration cannot pick traces by outcome, cannot bound spans per trace, and does nothing for a client that never fetches it.

open as a page

What access patterns must a Jaeger trace store support, and how do its Cassandra and Elasticsearch backends meet them?

level: seniorimportance: nice to knowfreq 26%

basics

~20 s

A Jaeger store must absorb write-heavy append-only ingest, answer point lookups by trace identifier, and serve ad-hoc search by service, operation, tag and duration. Cassandra needs extra index tables for that search; a search engine indexes everything at ingest instead.

open as a page