An OpenLLMetry service shows workflow spans but no LLM spans. How do you diagnose it?

## Read the evidence before you start The detail that makes this tractable is *which* spans are missing. If nothing at all arrived, the suspects are init ordering, exporter destination and process lifetime. Here decorator spans do arrive, which means `Traceloop.init()` executed, a tracer exists, the exporter has a reachable destination, credentials are accepted and spans are being flushed. A candidate who starts by checking the endpoint has not read the evidence. So the failure is in *auto-instrumentation*: the provider call is not producing a span in the first place. ## Cause 1: the instrumentation package is not installed Each library is traced by its own `opentelemetry-instrumentation-*` distribution. If that distribution is missing from the runtime image, that library is simply untraced — no exception, no warning in the application. This happens more than it should: a slimmed production image, a dependency group excluded from the install, a lockfile that resolved differently on the build machine. Check what is actually installed in the running container, not what the dependency file says. ## Cause 2: the library is excluded at init `Traceloop.init()` accepts `instruments` (an exclusive allowlist) and `block_instruments` (a subtraction). An allowlist written a year ago will silently exclude a provider added last month, and a temporary block from an incident can outlive the incident. Read the actual `init` call and any environment-specific config that builds it — this is a one-line cause that can burn an afternoon because nobody thinks to look at the config that was "already working". ## Cause 3: client version drift Instrumentors wrap specific methods on specific classes of the client library. A major upgrade can rename, move or restructure those code paths, and the patch then finds nothing to wrap. The tell is timing: tracing worked, then a dependency bump landed, then LLM spans stopped — with no application error, because instrumentation is designed to fail quietly rather than break the app. Check the client version against what the instrumentation release supports, and pin while you wait for a compatible release. The same shape of failure appears when the client is imported and used in a way that bypasses the patched entry point. ## Cause 4: the call does not go through the instrumented library Coverage is per known client, not per network call. If your service reaches the model through raw `httpx`, a bespoke internal wrapper, an unsupported SDK, or a self-hosted inference endpoint, there is no GenAI span to create — you may see a generic HTTP span if HTTP instrumentation is present, which some people misread as "tracing is on, LLM spans are broken". Similarly, a framework may call the provider along a path the framework instrumentation does not cover. The fix here is not debugging: it is wrapping that call in your own `@task`-decorated function or adding manual instrumentation. ## The verification loop Do not iterate in production. Reproduce with the smallest possible script: `Traceloop.init(app_name="probe", exporter=ConsoleSpanExporter(), disable_batch=True)`, one model call, run it. Two outcomes: - **A model span prints.** Instrumentation works in that environment, so the difference is environmental — the deployed image, its installed packages, its `init` arguments, or the code path production actually takes. - **No model span prints.** You have reproduced the fault in the smallest scope. Now bisect: pin the client library to its previous version, check the installed instrumentation package, and remove any instruments arguments. A console exporter with immediate export is the right tool because it removes batching, network and backend from the experiment entirely. ## Ordering, and why it is usually a red herring here A popular guess is "init ran too late". It is worth ruling out — calls made before `init` are never traced — but it rarely explains a *steady-state* production symptom, because a service that keeps serving requests will have initialized long before the traffic you are looking at. Ordering explains missing spans at startup, in tests, and in scripts. If spans are missing for every request all day, look at patching instead. ## What good looks like in the answer Name the split first (export proven, patching suspect), then the four causes, then the console-exporter reproduction. Add one preventive measure: a smoke check on deploy that asserts a model span was produced, so a silent instrumentation regression from a dependency bump is caught by CI rather than discovered weeks later when someone asks why the cost dashboard flatlined.