What access patterns must a Jaeger trace store support, and how do its Cassandra and Elasticsearch backends meet them?

## The three access patterns Jaeger's storage layer looks unusual until you write down what actually reads and writes it. There are exactly three shapes, and they pull in different directions. 1. **Write-heavy, append-only ingest.** Spans arrive continuously, individually, out of order, from many processes. They are never updated and never deleted individually. Two spans of the same trace may arrive seconds apart and are not joined on the way in — a trace is a *read-time* construct. 2. **Point lookup by trace identifier.** Opening a trace in the UI means "give me every span carrying this trace ID". This is the cheapest possible query if the trace ID is the key, and ruinous if it is not. 3. **Multi-predicate search.** The search screen filters by service, operation name, tag key and value, a minimum and maximum duration, and a time range, then returns a bounded list of matching traces. This is an ad-hoc query over several independent dimensions — and it is the pattern that decides which backends are viable. Pattern 3 is the awkward one. Patterns 1 and 2 are satisfied by almost any key-value store; a system that also has to answer "spans from `order-api`, operation `reserveStock`, tag `error=true`, longer than 250 ms, in the last hour" needs real indexing. ## What that demands of the backends | | Wide-column store (Cassandra) | Search engine (Elasticsearch / OpenSearch) | |---|---|---| | Ingest | Very cheap; append-friendly writes are its strength | More expensive; every span is indexed as it lands | | Lookup by trace ID | Direct, keyed read — the natural case | A term lookup on the trace ID field | | Ad-hoc multi-field search | Not native: Jaeger writes extra index tables per searchable dimension | Native: what an inverted index exists to do | | Cost of a new search dimension | Another write on every span | Usually none — already indexed | | Retention | Time-to-live on the written rows | Delete whole indices, which are written per day | The row that matters is the third. A wide-column store answers queries against keys you declared in advance, so Jaeger cannot simply ask it for "all spans with this tag longer than that duration" — it maintains additional index structures keyed by service, by service and operation, by duration, and by tag, and writes into them alongside every span. That is real **write amplification**: one span produces several writes, storage grows faster than the raw span volume suggests, and tag search is coarser than it looks because only indexed tags are searchable. In exchange you get a backend that shrugs at ingest rates a search cluster would struggle with. A search engine inverts the trade. Search is natural and expressive because the whole document is indexed, but you pay for that indexing on every single span at ingest time, and the cluster you must operate to absorb the write rate is heavier than the equivalent wide-column ring. Jaeger writes its data into per-day indices, which makes retention a matter of deleting whole indices on a schedule rather than expiring individual rows — cheap and predictable, at the price of day-granularity retention. ## Retention, which is where the bill lands Traces are the highest-volume signal most estates produce, and nobody keeps them long. The two mechanisms differ in an operationally important way: - **Row-level expiry** (the wide-column path) is set when the span is written, so changing retention only affects data written afterwards, and expired data still occupies space until the store reclaims it. - **Index-level deletion** (the search-engine path) drops a whole day at once, which reclaims space immediately and predictably, but means your retention granularity is a day and a single oversized day is deleted whole. Either way, the sizing question is the same one: sampled traces per second, multiplied by mean spans per trace, multiplied by mean span size, multiplied by retention — and then multiplied again by whatever the index overhead is. In a seed-catalogue ordering estate where one team contributes seventy per cent of the span volume, that multiplication is also the argument for setting that team's sampling rate differently from everyone else's, because storage is the constraint the sampling policy is really being tuned against. ## The rest of the options Jaeger also ships small backends that are not production storage and should not be pitched as such: an in-memory store used by the single-binary distribution, and an embedded local key-value store useful for a single node or a development box. Neither is horizontally scalable. Beyond that, Jaeger exposes a storage interface over gRPC so a backend it does not ship can be plugged in without forking it — which is how vendor-specific and object-storage-backed trace stores attach. The judgement an interviewer is really testing: can you name the read patterns before naming a product? A candidate who says "Elasticsearch, because search" without being able to say what is being searched has memorised a deployment, not a design.