Explain the built-in trace samplers in the OpenTelemetry SDK — AlwaysOn, AlwaysOff, TraceIdRatioBased and ParentBased. Which is the default, what information does a sampler see when it runs, and what do the OTEL_TRACES_SAMPLER and OTEL_TRACES_SAMPLER_ARG environment variables set?

## What sampling decides, and when Head sampling is the SDK deciding, at the instant a span starts, whether it will be recorded and exported. The decision has three possible outcomes: - **DROP** — the span is not recorded; it still gets a valid span context so children and downstream services can be parented and the trace stays structurally coherent. - **RECORD_ONLY** — recorded in-process (visible to anything reading spans locally, e.g. deriving metrics from them) but not exported. - **RECORD_AND_SAMPLE** — recorded and passed to the processors for export. The sampler's inputs are fixed: parent context, trace id, name, kind, initial attributes, links. It cannot see duration, status, or anything that happens later — which is the structural reason head sampling can never mean 'keep all the errors'. ## The built-ins **AlwaysOn** samples everything. It is the right default while a system is small, and it is what the SDK's default wraps. **AlwaysOff** samples nothing. Useful to silence a service without removing instrumentation, keeping propagation intact. **TraceIdRatioBased(p)** derives the verdict deterministically from bits of the trace id compared against a threshold. Two properties matter. First, the *same trace* yields the same verdict everywhere, so services configured with the same ratio agree — sampling is consistent without any coordination. Second, it is uniform over trace ids, not over customers or endpoints: 1% of everything, including 1% of your rare endpoints. **ParentBased(root, …)** is a wrapper, not a decision. If the span has a parent, it honours that parent's sampled flag (with separately configurable behaviour for remote-sampled, remote-not-sampled, local-sampled and local-not-sampled parents); if there is no parent, it consults the delegate. This is what makes traces all-or-nothing rather than a scatter of unrelated fragments: exactly one service — the first one — decides, and everyone else complies. The SDK's default sampler is ParentBased(AlwaysOn). Read literally: honour the incoming decision, and if you are the root, keep it. ## Configuring it Environment configuration uses two variables: - `OTEL_TRACES_SAMPLER` — a name: `always_on`, `always_off`, `traceidratio`, `parentbased_always_on`, `parentbased_always_off`, `parentbased_traceidratio`, plus implementation-specific options such as a remotely-configured sampler or vendor samplers. - `OTEL_TRACES_SAMPLER_ARG` — the parameter for the chosen sampler; for the ratio samplers a number in [0,1] (`0.05` = 5%), for remote samplers a small key/value string carrying an endpoint and poll interval. Using a bare `traceidratio` rather than the parent-based variant is a classic mistake: each service then decides independently, and although the trace-id derivation makes them *agree* when the ratio is identical, any service configured with a different ratio produces partial traces. Wrap in ParentBased and vary only the root. ## The trade-off Dropping is what makes tracing affordable, and what it costs is coverage: at 1% the trace you want during an incident probably does not exist. Mitigations belong to different layers — biasing rates by route or importance via a custom sampler, or deferring the decision to a collector that can see the finished trace. Both are strategy decisions; the mechanics above are what you configure in the SDK. ## Custom samplers The interface is small and implementing it is legitimate: keep everything on a low-traffic critical endpoint, drop health checks entirely, sample by a start-time attribute. Two rules: it must be fast, because it runs on every span start; and it must be deterministic with respect to the trace id if you want consistency across services. ## What sampling is not A dropped span still carries a valid context and still propagates. Sampling is not a security or privacy control, it is not a rate limiter for your backend's ingestion (the exporter and processors handle back-pressure), and it never removes attributes from spans that were kept.