Per-field GraphQL tracing emits thousands of spans per document; how do you afford that in production?

## Where the volume comes from The span count of a per-field trace is not proportional to the size of the document. It is proportional to the size of the **result**, because the executor resolves each selected field once per parent object it is resolved against. A ticketing read that walks a seat map for one performance selects perhaps a dozen fields, and resolves 14 sections, 644 rows and 1,847 seats; at four selected fields per seat that is over 8,000 field resolutions and therefore over 8,000 spans, for what a user sees as one page. Add a list of upcoming performances to the same document and it multiplies again. That is the shape of the problem: not a few expensive spans, but an enormous number of nearly free ones. And the export cost, the storage cost and the query cost of a tracing backend all scale with span count, not with how interesting the spans are. ## Lever one: the sampling unit is the request The granularity of the keep-or-drop decision is the whole operation. A trace missing an arbitrary 90% of its fields is not a cheap trace, it is a useless one — the parent-child structure is broken, self times cannot be derived, and the field you are hunting is probably one of the missing ones. The decision is taken once, at the start of the request, and every span inside inherits it. The GraphQL-specific refinement is what you key the rate on. A flat rate across one endpoint is dominated by whatever operation runs most often, and the rare expensive report query — which is exactly the one you want traced — shows up in the sample almost never. Keying the rate on the operation identity (the document hash is the safe key; a client-supplied operation name is unbounded and forgeable) lets you sample the hot lookup at a very low rate and the rare heavy document at a very high one. A complementary rule is to buffer spans for the request and always emit when the request was slow or produced field errors, which costs memory per in-flight request in exchange for keeping the traces that matter. ## Lever two: most fields do not deserve a span Most fields in a large result are resolved by the default behaviour: look up a key on the parent object and return it. That resolution is nanoseconds of real work; wrapping it in a span costs a clock read, an allocation and an export payload, so the instrumentation is orders of magnitude more expensive than the thing it is measuring, and it contributes nothing but noise to the trace. Two filters work well. Instrument only fields whose resolver is not the default one — the server knows which those are when it builds the executable schema, so this is a build-time decision, not a per-request check. Or record intervals for everything but only *emit* spans above a threshold, keeping the sub-millisecond ones folded into their parent's self time. The first is cheaper and coarser; the second catches the field that is usually trivial and occasionally is not. ## Lever three: collapse the siblings The 1,847 spans on `performance.seatMap.sections.*.rows.*.seats.*.status` differ only in their list indices. Their individual identities almost never matter; their distribution always does. Emitting one span per sibling set, carrying a count, a total, a maximum and perhaps the path of the slowest member, keeps every question you actually ask of the data and removes three orders of magnitude of volume. This is also the fix for the read-the-numbers trap where a shared batched fetch parks its whole wait on one arbitrary sibling: the aggregate shows the true total for the set. ## Lever four: aggregates for the steady state, spans for the investigation Spans are for answering "why was *this* request slow". They are a poor substrate for "is `Row.seats` getting slower over the quarter". The distinction that matters is the key: a response path with list indices in it is an unbounded key, and no store will thank you for grouping on it, whereas the **schema coordinate** — the parent type and field name, `Row.seats` — is bounded by the size of the schema. Per-coordinate duration aggregates can run at 100% on every request at a fraction of the cost of tracing, and the sampled field traces exist to explain what the aggregates surface. ## Do not forget the on-CPU cost Even a span you throw away costs you. The tracer sits inside the executor's per-field path, so it pays a clock read and usually an allocation for every one of those 8,000 fields. Benchmark the same document with instrumentation disabled versus enabled-and-dropped; if the delta is significant, the fix is the trivial-field filter, not a lower sample rate, because sampling reduces export and storage but not the per-field work of deciding. ## And it compounds across services In a supergraph composed from 62 subgraphs, every subgraph is resolving fields and every subgraph is capable of tracing them, so one client operation can fan out into per-field traces in a dozen services at once. The rule that saves you is that the sampling decision is made once for the client operation and carried, so the whole distributed trace is kept or dropped together — independent per-service sampling produces a large volume of partial traces, which is the worst of both outcomes. ## What a strong answer sounds like Count the spans before proposing a fix; sample by request and key the rate on operation identity; stop instrumenting fields that do no work; aggregate sibling list items; keep bounded per-coordinate aggregates always on and reserve full field traces for a sample and for slow or errored requests; and measure the instrumentation's own overhead, not just the backend bill.