How does an AWS X-Ray sampling rule decide whether a given request is traced? Explain the reservoir and fixed-rate fields, how rule priority works, and how one reservoir is shared across many instances of the same service.

## Why the two knobs exist A plain percentage is a bad sampling policy at both ends of the traffic range. At 5% a low-traffic endpoint that gets ten requests a minute may produce no traces at all for hours — exactly the endpoint you most want examples of. At 5% a hot endpoint doing 20,000 requests per second produces 1,000 traces per second, which is expensive and adds nothing over 50. X-Ray's answer is a **reservoir plus a rate**: - `ReservoirSize` — a fixed number of matching requests **per second** that are always traced. This is the floor: quiet endpoints still yield examples. - `FixedRate` — a fraction between 0 and 1, applied to matching requests *after* that second's reservoir is used up. This is the ceiling-shaper: busy endpoints scale proportionally rather than linearly with traffic. `ReservoirSize: 1, FixedRate: 0.05` — the values of the built-in default rule — reads as "always give me one trace a second, plus 5% of everything else". ## Matching and priority A rule matches on the fields of the incoming request as the sampler sees them: ```json { "RuleName": "checkout-post", "Priority": 100, "ReservoirSize": 5, "FixedRate": 0.10, "ServiceName": "checkout-api", "ServiceType": "*", "Host": "*", "HTTPMethod": "POST", "URLPath": "/orders*", "ResourceARN": "*", "Version": 1 } ``` `*` is a wildcard and `?` matches a single character, so most rules are mostly wildcards with one or two meaningful matchers. **`Priority` orders evaluation, lowest number first, and the first matching rule wins** — later rules are not consulted, so a broad rule with a low priority number will shadow every specific rule beneath it. That is the single most common configuration mistake: someone adds a catch-all at priority 1 to "trace everything for a day" and every carefully-tuned rule below it goes dead. The **default rule** always exists, cannot be deleted, and sits below everything else. If you never configure anything, that is what you are running. ## How one reservoir works across a fleet The interesting part. `ReservoirSize: 5` means five traces per second *for the service*, not five per instance — otherwise a fleet that scaled from 4 to 400 tasks would silently multiply its tracing bill by a hundred. Making that work requires coordination, and X-Ray does it centrally: 1. Clients (the SDK's sampler, or the daemon acting on their behalf) call `GetSamplingRules` to fetch the rule set, and refresh it periodically so console changes take effect without a deploy. 2. Clients then call `GetSamplingTargets` on a short cycle, **reporting how many requests they saw and sampled**. 3. X-Ray divides the reservoir across the reporting clients in proportion to the traffic each reported, and returns each one a **quota** valid until an expiry time. 4. A client that has just started and has no quota yet may **borrow** — one trace per second — so a fresh instance is never blind while it waits for its first target. When a client cannot reach the sampling API at all, it falls back to local rules rather than dropping tracing entirely. The practical consequences: rule changes propagate on the order of minutes, not instantly; reservoirs are only honoured approximately during rapid scaling events, because quota allocation lags traffic; and a service whose instances see wildly uneven traffic will see the reservoir concentrate on the busy ones, which is usually what you want. ## Where sampling actually happens The decision is made **once, at the entry point of the trace**, and then propagated in the `Sampled` field of the trace header. Downstream services honour an incoming decision rather than re-deciding. That is what keeps a sampled trace complete end to end — sampling per service would produce traces with holes in the middle, which are worse than no trace at all. It also means the rule that matters for a given request is the rule evaluated by the *first* instrumented component in the chain. Tuning a rule on a downstream service will not change how often it appears in traces started upstream. ## Tuning it in production Good defaults: leave the default rule as a safety net; add rules with a healthy reservoir for low-traffic but high-value paths (checkout, auth, anything with a customer-facing SLO); add a rule with a low or zero rate for the endpoints that flood without informing — health checks especially, which can otherwise dominate your traces and your bill. And when you raise sampling to chase an incident, raise it on a *specific* rule with a *specific* priority, and put a calendar reminder on lowering it again.