In LangSmith, how do you split coverage between an online judge and human review?

## Two currencies, not one budget The judge costs money and scales instantly. Human review costs attention from people who have other jobs, scales badly, and cannot be bought at the last minute. Treating them as a single "quality budget" leads to the classic mistake of trading reviewer hours for sampling rate as if they were fungible. They buy different things: the judge buys *coverage*, humans buy *credibility*. ## Assign each to what it is good at **The judge goes wide and shallow.** A low sampling rate across all root runs of the application, a cheap fast model, reference-free criteria. Its output is a continuous line you can look at daily. Its job is detection: something changed, or nothing did. **Humans go narrow and deep.** A short list of slices where a wrong answer is expensive or where you need a label you can trust: - the first week of any newly deployed prompt or model version - flows with regulatory, financial or safety consequences - runs already carrying a low user score, an error, or a low judge score - runs where the judge and the user disagreed — the most information-dense sample in the whole system - a small unbiased random slice, deliberately kept, so you notice failure modes your filters were not designed to catch That last one is easy to drop and expensive to have dropped, because every other filter only finds problems you already anticipated. ## Implement as several rules One global rule at one rate is a compromise nobody chose. In practice you have three or four: a judge rule at a low rate over everything; a judge rule at a much higher rate scoped to the new version; an annotation-queue rule filtered on bad feedback; and a small random-sample queue rule. Each has its own cost line and its own justification, and each can be turned off independently when it stops earning its keep. ## The one dependency between them Human labels are the calibration set for the judge. If the judge writes a score on the same runs a reviewer grades, you can compare the two on that overlap. When they agree, the judge's wide, cheap series is meaningful and you can act on it. When they do not, the series is decoration, and every decision made from it is unfounded. So deliberately keep an overlap — do not construct the rules so that judged runs and reviewed runs are disjoint sets, which is what happens by accident if the queue is fed only by runs the judge scored low. This check is not one-off. It has to be repeated whenever the judge's prompt or model changes, whenever the application changes materially, and on some periodic cadence, because a judge that tracked reality in March can quietly stop doing so. ## Size the human side from real hours Do the arithmetic honestly. Reviewers get through some tens of substantive runs an hour. Multiply by the hours the organisation has genuinely committed per week. That is your queue's intake, and the feeding rule's sampling rate follows from it — not the other way round. The characteristic failure is a rule set at an aspirational rate, a queue that accumulates thousands of pending runs, and a team that stops opening it while continuing to describe human review as part of their process. Who reviews matters as much as how many. Labels from a domain expert are worth calibrating against; labels from whoever was free are noise with a timestamp. If the rubric needs domain knowledge, engineers are the wrong reviewers and the queue instructions must be written for the people who are right. ## What each layer is allowed to decide Be explicit, because this is where teams get burned: - **Offline experiments over a curated dataset gate a deploy.** They are repeatable, they have references, and they run before anything reaches a user. - **Online judge scores detect**, over hours and days. Traffic mix shifts underneath them, sample sizes are modest, and they cannot distinguish "the model got worse" from "the questions got harder". Never gate on them alone. - **Human review confirms**, and produces the corrections that become tomorrow's dataset examples. A regression discovered by the online judge should end its life as a dataset example added via the annotation queue, so the same failure is caught before deploy next time. If it does not, you will keep rediscovering it in production, which is a process defect rather than a tooling one. ## Making the spend defensible Expect to justify this. The useful framing is per-incident: what did an escaped quality regression cost last time in support load, customer churn or remediation, and what fraction of that is the evaluation budget. Coverage should also be proportional to blast radius — a high-volume, low-stakes summarisation feature does not deserve the same review intensity as a low-volume flow that can commit the company to something. Finally, review the whole arrangement periodically. Rules accumulate. A judge rule created for a launch six months ago is still sampling and still billing, and a queue that no longer matches how the product works is producing labels against an obsolete rubric.