You must choose a trace sampling strategy for a fleet of services that call one another. How do you keep the decision consistent across a whole trace, how do you decide between deciding at the entry point and deciding later in a collector, and how do you set the rate?

## The two questions that actually matter Sampling strategy reduces to *where* the decision is made and *what it is allowed to know when it is made*. Everything else is arithmetic. ## Consistency is a correctness property A trace sampled by half its services is worse than one not sampled at all: it looks complete, and the gap is indistinguishable from a real fault. The mechanism that prevents it is parent-based sampling everywhere, so the decision travels with the context and only the root decides. That leaves the roots. A fleet usually has several: each ingress service, plus every asynchronous entry point — a scheduled job, a consumer of an external queue. All of them must use the same rate and the same trace-id-derived algorithm; deterministic derivation is what makes independent processes agree without coordination. Two ingress tiers with different rates yield partial traces whenever a request crosses between them. A related nuance: because the derivation depends on the trace id, sampling is uniform over traces, not over customers, endpoints or error rates. Everything you care about that is rare is rare in the sample too. ## Head versus post-hoc **Head** (in the SDK, at the first span): the decision is made before the outcome exists. Its advantage is economic and structural — dropped spans are never serialised, never sent, never received, so cost falls at the source and no component needs to buffer. Its limit is absolute: you cannot ask for 'all traces with an error' because at decision time there is no error. **Post-hoc** (in a buffering collector, after the trace is assembled): the decision sees the finished trace and can keep every error, every latency outlier, and a small sample of the boring remainder. That is exactly the population humans want. It costs: every span must be transmitted and received even though most are discarded, the buffering layer must hold traces for a decision window (so late spans past that window are missed), and all spans of a trace must reach the same decision instance, which constrains how you scale and route the collection tier. Configuring that tier belongs to the collector layer; the strategic choice is yours. A common composition is a coarse head sample to cut the obvious bulk, plus a post-hoc layer for the surviving population. Reason about it as a multiplication — the head rate bounds what any later stage can possibly keep — and remember that once head sampling has dropped a trace, no downstream cleverness can bring it back. ## Setting the rate Work from a budget, not a habit: 1. Take the ingestion or storage allowance you can defend and divide by traffic to get a baseline rate. 2. Bias per route. Health checks and readiness probes deserve zero. A low-traffic, high-value endpoint can stay at 100% and cost nothing. A hot endpoint at 1% still yields plenty of examples per minute. 3. Check absolute counts, not just percentages: a rate is only adequate if it produces enough traces per minute on your *rarest important* path to be useful during an incident. This is where uniform ratios fail and per-route rules earn their complexity. 4. Keep development and staging at 100% — volume is trivial and the debugging value is highest. ## Custom root samplers Route- or attribute-aware decisions require a custom sampler at the root, using only start-time inputs. Two constraints: it must be fast, since it runs on every span start, and it should stay deterministic in the trace id so it remains consistent if more than one service ever roots a trace. Keep the rules few and centrally owned, because sampling rules are effectively a query the fleet has committed to answering, and per-team drift makes fleet-wide comparisons meaningless. ## What sampling must not be used for Not a privacy control — kept traces keep everything. Not back-pressure — that is the processors' and exporters' job. Not a substitute for instrumentation coverage — a broken hop is missing at 100% sampling too. And not something to change quietly: a rate change silently rewrites the denominator of anything computed from traces, so any span-derived metric must be produced before sampling or explicitly scaled. ## The observable you owe yourself Whatever you choose, publish the effective rate as telemetry, and monitor traces kept per minute on the paths you care about. Sampling policy drifts, gets overridden per service, and is discovered to be wrong exactly when you are trying to use it.