Agent vs Library Instrumentation

## What "automatic" means mechanically An automatic instrumentation agent is code loaded into the application process before or alongside your own code: a runtime agent that rewrites classes as they load, an import hook that wraps functions, a preloaded shared library, or a runtime switch that enables hooks already compiled in. The mechanism differs per language; the shape does not. **The agent carries a catalogue of instrumentations, each matching a named library within a version range.** When a matching type or function is loaded, the agent wraps it — entering starts a span, leaving ends it, and the error path records the failure on it. Everything else follows from that catalogue. Coverage is exactly the set of libraries somebody wrote a plugin for, running at versions those plugins still match. Nothing in the word *automatic* implies the agent understands your program. ## What it reliably gives you - **A server span per inbound request** — protocol, route template, status, duration. - **A client span per outbound call it recognises** — HTTP and RPC clients, database drivers, cache clients, message producers and consumers. - **Context plumbing** — writing trace-context headers onto outbound requests, reading them off inbound ones, and following the runtime's thread pools, executors and callback machinery so the active trace survives a hand-off. - **Errors at those boundaries** — the exception that escaped a hooked call, attached to the span that was open when it escaped. - **Process identity** — host, runtime, version, plus whatever service name you configure. That is not a small gift. It is uniform across services, it costs no engineering time per service, and it is the only realistic way to get a baseline on a fleet nobody has time to instrument by hand. ## What it is structurally blind to | Blind spot | Why the agent cannot see it | | --- | --- | | Your domain logic | No library call is crossed, so no hook fires | | An in-house transport or framework | No plugin was ever written for it | | Which branch ran, and why | The hook sees the call, not the decision behind it | | Business identity (tenant, plan, cohort) | Those values live in your objects, not in the call signature | | A library version outside a plugin's match range | The match fails silently — no error, simply no spans | | Time below the hook point, such as lock waiting | The hook measures only the call it wrapped | The last two bite hardest. A silent match failure is indistinguishable from "that code never ran", and any time not spent inside a hooked call simply disappears into the parent's duration. ## What that does to the shape of the trace Take a school-meal ordering service at its 5,400-request-per-second lunchtime peak. Agent-only, one order request renders as a server span of 847 ms containing three children: two database calls of 14 ms and 11 ms, and one outbound call to an internal payments service of 63 ms. Eighty-eight milliseconds are accounted for. The other 759 ms are not. The agent is telling the truth. The request did take 847 ms and 88 ms of it was I/O the agent recognises. What it cannot tell you is that 612 ms of the remainder went into evaluating eligibility rules, because no call site in its catalogue was crossed while that ran. The trace is **correct at the boundaries and opaque in between** — shaped like your I/O rather than like your program. Engineers misread that gap in two opposite directions. Some conclude the trace is broken; it is not, it is complete for what it covers. Others conclude the service was waiting on something invisible; it was not, it was running. The right reading of a large difference between a span's duration and the sum of its children — its **self time** — is "the answer lives in code this agent does not instrument". ## Diagnosing before you blame the agent 1. Compare each span's duration with the sum of its children. A large self time is a pointer, not a fault. 2. Check what the agent actually installed. Agents report which instrumentations loaded; a plugin that did not match will be absent from that report. 3. Check library versions against the agent's supported range, especially straight after a dependency bump. 4. Confirm the work is genuinely in-process. An outbound call made over a raw socket, with no recognised client library, also surfaces as self time. 5. Only then add instrumentation, and add it exactly where the gap is rather than everywhere. ## The costs you are accepting Class rewriting or import patching happens at startup, so start-up time grows measurably and start-up failures become a new class of incident. Every span costs allocation, serialisation and network bytes: a service emitting six spans per request at 5,400 requests per second is producing 32,400 spans per second before any sampling decision is taken. And an agent is third-party code running inside your process — a plugin interacting badly with a library version can change exception behaviour or timing. Roll an agent out the way you roll out a deploy: canary first, with a switch that turns it off.