Explain what an exemplar attached to a Prometheus or Mimir time series is, everything that has to be enabled end to end for one to appear as a clickable dot on a Grafana time series panel, and why those clicks sometimes lead to a trace that no longer exists.

## What an exemplar is Metrics are aggregates: a histogram tells you how many observations fell in each bucket, not which ones. That is exactly what makes metrics cheap and exactly what makes them useless for root cause. An **exemplar** is a small escape hatch built into the metric: alongside a sample, the instrumentation may attach one example observation with its own labels and its own timestamp. The canonical labels are `trace_id` (and often `span_id`). Semantically: *this bucket got incremented, and here is one request that did it.* The payoff is the classic RED-metric workflow. A panel shows p99 latency spiking at 14:32. Without exemplars you now go hunting in the trace store for a slow request near 14:32 for that service, which is guesswork. With exemplars you click the dot sitting on the spike and land in a trace that *provably* contributed to it. ## The four links of the chain **1. Instrumentation.** The library recording the observation must support exemplars and must have access to the current trace context at record time. Typically only histogram observations and counter increments carry them, and only when a trace is active. **2. Exposition.** In the pull model, exemplars require the **OpenMetrics** exposition format: the exemplar follows the sample on the same line, after a `#`, as a label set plus a value and optional timestamp. A scraper must negotiate OpenMetrics for them to survive. In the push model they travel in OTLP alongside the data point. **3. Storage.** In Prometheus, exemplar storage is **opt-in** — a feature flag enables it, and a configuration value sets the size of a **circular in-memory buffer**. Two consequences follow and both matter in interviews: exemplars are **not persisted to the TSDB blocks the way samples are**, and the buffer **overwrites oldest-first**. High-churn services can evict exemplars within minutes, so historical panels show none. Mimir applies its own per-tenant exemplar limits, which default to zero in some deployments — a very common reason nothing ever shows up. **4. Grafana.** The Prometheus/Mimir data source has an **Exemplars** section: add an entry, choose *internal link*, select the trace data source (stored as a **UID**), and set the **label name** that holds the id, usually `trace_id`. Then the panel: exemplars render only on the **time series** panel, and the query option **Exemplars** must be enabled. Grafana requests exemplars only for range queries; an instant query returns none. They appear as small diamonds on the x-axis band; hovering shows the labels, clicking follows the link. ## Why clicks dead-end - **Sampling.** This is the big one. If the tracing pipeline uses head sampling at, say, 1%, then 99 of every 100 exemplars point at a trace that was never exported. The exemplar records the id regardless — the metric path does not know the trace was dropped. Mitigations: sample tail-based on latency or errors so the interesting traces (exactly the ones exemplars point at) are always kept, or raise sampling for the services where this workflow matters. - **Retention asymmetry.** Metrics are cheap and kept for months; traces are expensive and kept for days. Every exemplar older than trace retention is a dead link by construction. Say so explicitly rather than treating it as a bug. - **Buffer eviction on the metrics side** — the mirror-image failure: the trace exists but no exemplar survived to point at it. - **Encoding mismatch** — the label carries the id in a different form than the trace store expects (dashes, wrong length, non-hex), so the lookup misses. ## How to pitch it The framing that separates a strong answer: exemplars are a **sampled, best-effort, bounded** pointer from an aggregate to an individual. They are not an index and were never intended to be complete — you cannot enumerate all requests in a bucket. Treat coverage as probabilistic, and design the sampling policy so that the traces exemplars point at are exactly the traces the tail sampler keeps. When those two policies are aligned the workflow feels magical; when they are not, the dots are decorative and users stop clicking them.