At high request volume, how do you decide which LangSmith runs to trace at all?

## Why sampling becomes a decision at all At low volume you trace everything and never think about it. Three things change that as traffic grows. **Cost**: ingest is billed and large-context RAG prompts are heavy, so a fully traced high-volume endpoint can outweigh the model spend it is observing. **Signal**: a project receiving millions of runs an hour is not more useful than one receiving thousands — it is less useful, because the interesting run is harder to find. **Exposure**: every traced request copies more content into a third-party store, and volume multiplies whatever privacy risk one request carries. So the question is not "can we afford it" alone; it is "what do we actually need to be able to reconstruct later". ## The two mechanisms **Global head sampling.** `LANGSMITH_TRACING_SAMPLING_RATE` takes a value from 0 to 1. The decision is made at the root run and applies to the whole tree, which is the property you want: a partially sampled trace, with the retrieval step present and the model call missing, would be worse than no trace. It is set per deployment, so different services or environments can carry different rates without code changes. **Programmatic per-request decisions.** `tracing_context(enabled=...)` wraps a block and switches tracing on or off for everything inside it. Because it is code, the predicate can use anything the request knows at its start: is this internal traffic, is this account on the canary prompt, is this a paying tier, is a debug flag set, did the previous turn in this conversation fail. Placed in middleware alongside the labelling context, it becomes the single place where the policy lives. **Project routing.** `project_name` on `tracing_context` or `@traceable` sends a class of traffic to its own project. This is underrated: separating "bulk production at 2%" from "canary at 100%" from "internal QA at 100%" means each project's monitoring charts describe a coherent population, and a filter over the canary project is not competing with a million background runs. ## The head-sampling limitation, stated honestly All of the above decide before the outcome is known. You cannot express "keep every trace that errored or exceeded three seconds", because at the moment of the decision the request has not run. This is the single most important thing to say out loud in an interview, because the naive answer — "sample 5% but keep all the failures" — is not something head sampling can deliver. What you do instead: - Keep **metrics and logs at full coverage**. Error rate, latency percentiles and token spend should come from your normal telemetry, which is cheap per event, not from the traced sample. Traces are for reconstructing individual executions. - **Bias the sample toward where failures live**: new prompt versions, new models, newly released code paths, accounts that have complained. A 100% rate on a canary that is 1% of traffic costs almost nothing and captures most of what you need. - **Make it reproducible.** If a specific request needs investigating and was not sampled, the fallback is replaying it with tracing forced on — which requires that you retained the inputs somewhere, and that replay is safe. If neither holds, raise the rate for that path instead. - **Allow a targeted override.** A per-request debug flag, or a rate you can raise at runtime without a deploy, converts a multi-hour investigation into a minutes-long one during an incident. ## Cost is mostly payload, not count A trace of a RAG request carrying twenty retrieved chunks is orders of magnitude larger than a trace of a short classification call. Before halving the rate — which halves your visibility — consider truncating or hiding the heavy fields on bulk traffic while keeping the run structure, timings and token usage. You retain latency analysis, error rates and cost attribution for 100% of requests and pay for content only where content matters. That is usually a better trade than dropping whole traces, and it is the move that separates a considered answer from a reflexive one. ## A workable default policy - Development and staging: 100%. - Internal, employee and QA traffic in production: 100%, own project. - Canary or newly deployed prompt/model paths: 100% until the rollout completes, own project. - Bulk production: a low single-digit percentage, content redacted or truncated, plus full-coverage metrics and logs. - An override switch that raises the rate for a named path or account without a deploy. Then revisit it, because the right rate is a function of current traffic, current spend and what your last three incidents actually needed — none of which are stable. ## The organisational half Someone has to own this. If each team sets its own rate against a shared budget, the budget is consumed by whoever traces most eagerly, not by whoever needs it most. Set the policy centrally — defaults per environment, a documented way to request more for a rollout, and a review when a project's volume changes materially — and make the default safe so that a team that thinks about none of this still ships something reasonable.