When an automatic instrumentation agent is already running, which code still earns a hand-written span?

## What the agent already drew An automatic instrumentation agent gives you a skeleton: a span where a request entered, a span for each recognised outbound call, durations and error status on both. That skeleton answers *which hop* was slow. It cannot answer *which decision* was slow, or *for whom*, because those facts live in your objects and never appear in the signature of a library call. So the question is not whether to add explicit instrumentation on top; it is what to add, and how to stop. Both failure modes are common and both are expensive: services with an agent and nothing else, whose traces bottom out in an unattributable gap, and services with three hundred hand-written spans that mirror the call graph and answer nothing. ## Three things only your own code knows - **The unit of work that means something.** The agent names spans after transport operations. Your code knows that the interesting unit is "evaluate eligibility", "reprice the week's menu", "generate the receipt" — the things whose duration you would actually act on. - **Identity.** Tenant, school, plan tier, route class, feature-flag variant, batch size, whether the request came from a retry. These make a trace *findable*. A trace you cannot find when you need it is a trace you did not collect. - **Outcome and decision.** A fallback was taken. A cache missed and the slow path ran. A validation rejected the order. Three attempts happened before one succeeded. The agent sees a successful HTTP 200; only your code knows which of four routes through the function produced it. ## The rule for what earns a hand-written span A new span is worth its cost when you can answer both of these: 1. **What incident question does it answer?** Name the question you have actually asked during an incident and could not answer. "Was the slowness in rules evaluation or in pricing?" is a question. "More visibility" is not. 2. **What decision changes with the answer?** If both possible answers lead to the same action, the span is decoration. Two supporting tests catch most of the rest. If the work is shorter than the noise floor of the surrounding measurements, the span adds bytes and no signal. And if the span would exist once per request per service, ask whether the same fact could be an attribute on a span you already emit. ## Span or attribute? | You want to know | Reach for | | --- | --- | | How long a distinct piece of work took | A span | | Whether that piece of work failed on its own | A span, so it can carry its own status | | Which tenant, route, variant or cohort a request was | An attribute on an existing span | | Which branch or fallback ran | An attribute on an existing span | | How many items a batch held | An attribute on an existing span | Attributes are dramatically cheaper. Adding one to a span you already emit costs bytes on a record already in flight, and unlike a dimension on an aggregated metric it does not multiply anything stored. Adding a span costs a whole new record, per request, forever. In a school-meal ordering service peaking at 5,400 requests per second, one extra span per request is 5,400 additional spans per second; one extra attribute is a few dozen bytes on records you are already paying for. ## What over-instrumentation looks like - Span names that mirror function names, so the trace reads like a stack rather than a story. - Traces with hundreds of spans that nobody expands, because the interesting one cannot be found. - Spans around getters, mappers and constructors added "for completeness". - Hand-written spans duplicating a boundary the agent already covers, so every database call appears twice. - Cost growing quarter over quarter with no new incident question becoming answerable. That last one is the honest measure. Instrumentation is an investment, and the return is measured in questions you can now answer under pressure. ## Keeping it from rotting Span names and attribute keys are an interface, not private detail — dashboards, saved queries and alert rules bind to them, and renaming one silently breaks things nobody will notice until an incident. Keep the names a service emits in one small module rather than as string literals scattered across the codebase, so the whole vocabulary of a service can be read on one screen. Give the handful that other teams depend on a test that fails if the key changes, so the rename is caught before it ships rather than during an outage. And put deletions on the same footing as additions: instrumentation that stopped answering a question is cost with no return, and removing it is legitimate work.