A tracing backend like Jaeger can't afford to store a full trace for every request in a system doing 500,000 requests per second. Compare head-based and tail-based sampling as strategies for deciding which traces to keep, and give a scenario where head-based sampling would make you miss the exact trace you needed.

## Why sampling exists at all Sampling exists because recording a complete trace, every span, on every service, for every single request, simply doesn't scale for high-traffic systems: at 500,000 requests per second, even a lean average trace of a few kilobytes would produce well over a terabyte of trace data per second, which is untenable to transport, store, and query, and mostly wasted, since the overwhelming majority of requests are unremarkable successes that nobody will ever look at. Sampling is the decision procedure for choosing which fraction of traces get fully recorded, and the fundamental design question is when that decision gets made. | | Head-based | Tail-based | |---|---|---| | Decision | at the root span, before any downstream work has happened and therefore before anything is known about how the request will turn out | deferring the decision until after the full trace's outcome is known | | Cost | the decision is made blind: a request about to hit a rare, severe bug has exactly the same probability of being sampled as a perfectly ordinary one | operational complexity and resource cost in the collection tier rather than in trace volume itself | ## Deciding at the root Head-based sampling decides at the root span, the very first service that starts the trace, before any downstream work has happened and therefore before anything is known about how the request will turn out. The simplest form is a fixed probability: flip a biased coin with, say, a 1% chance of sample. That decision is then propagated to every downstream service via the sampled flag in the W3C trace-context header, so every service in the call chain either records all its spans, if sampled, or records none, if not, keeping the recording decision consistent across the whole distributed call graph without any service needing to coordinate with any other. The mechanism is deliberately simple: - no service needs to buffer anything, - no collector needs to hold spans waiting for a decision, - and the system's storage and throughput cost is a predictable, fixed fraction of total traffic, which makes head-based sampling trivial to reason about and to scale. ## The blind spot The cost of that simplicity is that the decision is made blind. A request that is going to be perfectly ordinary has exactly the same probability of being sampled as a request that is about to hit a rare, severe bug, time out after ten seconds, or fail with an error in the payments service. At 1% sampling, a bug that occurs in 1 out of every 100,000 requests has only about a 1-in-10,000 chance of ever being captured in a trace at all, which for a genuinely rare but important failure mode means it may effectively never show up in the tracing system, even though it's exactly the kind of thing an engineer would want a trace for. ## Deciding after the outcome Tail-based sampling fixes this by deferring the decision until after the full trace's outcome is known. Instead of deciding at the root, an intermediary component, usually a tracing collector configured to buffer, holds all the spans belonging to one trace ID for some completion window, long enough for the slowest expected request to finish, and only then applies rules such as - keep if any span had an error status, - keep if total duration exceeded 500ms, - or keep a random 1% of everything else as a baseline sample. This guarantees that the traces most valuable for debugging, the errors and the slow outliers, are essentially always captured, regardless of how rare they are, while still controlling overall storage cost by discarding the bulk of unremarkable, fast, successful traces. ## What deferral costs The trade-off is operational complexity and resource cost in the collection tier rather than in trace volume itself. Every span from every service, sampled or not, has to be transmitted to the buffering collector, because you can't know in advance which traces will turn out to deserve keeping; the discarding only happens after the fact. That means the network and collector-ingestion cost looks a lot like 100% sampling even though final storage looks like a small fraction, and the collector needs enough memory to buffer every in-flight trace for the length of the completion window, which is itself a scaling and tuning problem, a window too short and slow traces get evaluated before they finish, a window too long and buffering cost balloons. Tail-based sampling also adds latency to the observability pipeline itself, since a trace isn't finalized in storage until its completion window has elapsed. ## The trace you would have missed A concrete scenario where head-based sampling causes a real miss: an intermittent bug causes 1 in 20,000 checkout requests to fail with a database deadlock, and the team runs head-based sampling at 0.5%. The odds that the specific request an angry customer reported, identified after the fact by its correlation ID, was also one of the ones randomly chosen for full tracing are roughly 1 in 200; in practice, the trace simply doesn't exist, and the on-call engineer is left reconstructing the incident from logs alone. With tail-based sampling and an always keep errored traces rule, that same deadlocked request would have been captured with certainty, because the decision was made after the collector saw that the trace ended in an error, not before. Production systems often combine both: head-based sampling at a modest rate to cap the raw ingestion volume the collector tier needs to handle, layered with tail-based rules on top of what does reach the collector, to bias what gets kept in final storage toward errors and outliers rather than a uniformly random slice.