Explain how tail-based sampling works in the OpenTelemetry Collector — the decision buffer, policies, and late-arriving spans — and what it costs compared with deciding at span creation time.

## Why the decision is deferred Sampling exists because storing every span of every request is unaffordable at scale. Head sampling makes the decision at trace start — a ratio applied to the trace id, propagated so the whole trace agrees. It is cheap, stateless and consistent, but the decision is made before the request has done anything: you cannot condition on "it failed", "it took four seconds", or "it touched the payments path", because none of that is known yet. Tail sampling exists to condition on exactly those things. ## The mechanism The `tail_sampling` processor keeps an in-memory map from trace id to the spans seen so far. A trace's clock starts on its first span. When no new span has arrived for `decision_wait` (default 30 seconds), the processor evaluates every configured policy against the assembled trace and produces a single verdict for the entire trace — sampling is all-or-nothing per trace, which is what makes the retained traces usable rather than full of holes. Capacity is bounded by `num_traces`, the maximum number of traces held simultaneously (default in the tens of thousands); `expected_new_traces_per_sec` pre-sizes internal structures. When the buffer is full, the oldest traces are evicted — which shows up as unexplained missing traces, not as an error, so it must be watched. ## Policies Each policy has a name, a type and its own parameters. The common set: - `status_code` — keep traces containing a span with `ERROR`. - `latency` — keep traces whose duration exceeds a threshold. - `string_attribute` / `numeric_attribute` / `boolean_attribute` — keep by attribute value, e.g. a specific tenant, endpoint or feature flag. - `probabilistic` — keep a percentage, the baseline for representative traffic. - `rate_limiting` — cap spans per second regardless of other policies, a safety valve. - `span_count` — keep unusually large traces (a fan-out gone wrong). - `ottl_condition` — arbitrary expressions over span or resource fields. - `and` / `composite` — combine policies; `composite` also lets you allocate a rate budget across sub-policies in priority order. Policies are evaluated as an OR by default: if any policy says sample, the trace is kept. That is why the typical configuration reads "all errors, all slow, plus 1% of everything else" — the union produces both complete failure coverage and an unbiased baseline. ## The three costs **Memory.** You are holding every span of every in-flight trace for the wait window. Budget roughly traces-per-second × `decision_wait` × average bytes per trace, and remember that attribute-heavy spans move that average a lot. This is why the sampling tier is sized on concurrent traces, not on request rate. **Latency to the backend.** Nothing appears in the backend until the wait expires. Thirty seconds of delay is fine for post-hoc debugging and wrong for a live tail view, which is why some teams run a separate always-on pipeline for a small unsampled slice. **Affinity.** The processor can only decide from what it holds, so **every span of a trace must reach the same Collector process**. Behind a normal load balancer that is false, and the failure is silent: policies evaluate against fragments. The standard remedy is a stateless routing layer whose `loadbalancing` exporter is configured with `routing_key: traceID`, hashing spans onto a consistent instance of the sampling layer. ## Late spans A span that arrives after its trace's decision has been made cannot retroactively change it. Depending on configuration the processor either drops such spans or applies the recorded decision for a limited retention period. Long-running traces — a job whose root span lasts minutes, a streaming request, a workflow spanning retries — are the pathological case: the trace looks idle, the window expires, a decision is taken, and the remainder is orphaned. Extending `decision_wait` multiplies the memory cost, so long-lived work is usually modelled as several shorter traces joined by span links instead. ## Interaction with derived metrics If you compute request-rate, error-rate and duration metrics from spans, compute them **before** sampling. Metrics derived after a 1% sampler describe 1% of your traffic; unless every consumer knows and corrects for the ratio — and error-biased policies make the ratio non-uniform — your dashboards are simply wrong. Ordering the pipeline as receivers → enrichment → span-metrics connector → `tail_sampling` → exporters gives accurate aggregates over 100% of traffic while storing a fraction of the traces. ## Choosing between head and tail Head sampling is right when the volume reduction is the whole point and uniform sampling is acceptable: it costs nothing, adds no latency, and needs no shared state. Tail sampling is right when the rare events are the valuable ones. Many estates run both — a modest head ratio to cut raw volume at the source, then tail policies on what survives — but be explicit that head sampling has already thrown away errors you will never see at the tail.