What is a LangSmith annotation queue, and how do runs end up in one?

## What it is for Automated scoring tells you a number moved. It cannot tell you what "wrong" looked like, and it cannot produce a trustworthy label. Human review does both, and it is expensive, so it needs to be structured rather than ad hoc. An annotation queue is that structure: a worklist of runs, a fixed rubric, and one reviewer at a time. Without it, human review in practice means someone filtering the runs table, reading a few traces, and forming an impression that never becomes data. The queue exists to make the reading produce feedback records. ## Getting runs into a queue Three routes, and a mature setup uses all three. **Automation rule.** The continuous route. A rule filters the project's runs and samples a fraction of them into the queue. This is how you get a steady, unbiased trickle of representative traffic to look at. **Manual selection.** From the runs table, after filtering to something interesting — errors, thumbs-downs, a specific customer's session, everything from the hour of an incident. This is the investigative route. **SDK.** `Client.create_annotation_queue(name=..., description=...)` creates one; `Client.add_runs_to_annotation_queue(queue_id, run_ids=[...])` pushes a list of runs into it. This is the programmatic route, used when your own logic decides what deserves review — for example, a nightly job that queues every run where a judge and a user disagreed. A useful pattern is chaining: a rule filters on *existing feedback* rather than raw traffic, so runs a judge scored badly, or runs the user thumbed down, are automatically escalated to a human. That gives you a queue full of probable failures instead of a queue full of ordinary successes, which is a very different use of reviewer time. ## The reviewing experience A reviewer opens the queue and is shown runs one at a time — inputs, outputs, and the trace underneath if they need to see which step went wrong. Runs are handed out individually rather than as a shared list, so two reviewers do not silently grade the same run and produce conflicting feedback on the same key; this is a property of the queue, not something you should be coordinating by convention in a chat channel. The queue carries the rubric: which feedback keys the reviewer fills in, what type each is (a number, a category, free text), and the instructions telling them what "good" means for this application. Keeping the rubric on the queue rather than in each reviewer's head is what makes two reviewers' output comparable. ## What a review produces The verdict is written as feedback on the run, under the queue's keys, recorded as human-sourced. That last part matters. If your judge writes `correctness` and your reviewers also write `correctness`, both series exist on the same runs, and you can compare them — which is how you find out whether the judge is worth trusting. If you cannot tell the two apart, you have destroyed exactly that comparison. The reviewer can also add a correction — what the answer should have been — and send the run to a dataset. That is the trace-to-dataset step: the run's inputs become the example's inputs, the correction becomes the expected output, and from then on every offline experiment you run includes this real failure. A queue whose reviewed runs never reach a dataset is generating opinions instead of assets. ## Capacity is the real constraint A judge scales with money; a queue scales with people. A reviewer might get through a few dozen substantive runs an hour, which sets a hard ceiling on the rule's sampling rate feeding it. The characteristic failure is a rule at a rate nobody sized, producing a queue with thousands of pending runs that everyone stops opening — worse than no queue, because it looks like coverage. Size the rate to your reviewers' actual weekly hours, and prefer a small queue of hard cases over a large queue of average traffic. Who reviews also matters. Domain experts produce labels worth calibrating a judge against; whoever was free that afternoon produces noise with a timestamp. If the rubric needs domain knowledge, engineers are not the right reviewers, and the queue's instructions have to be written for the people who actually are. ## Privacy A queue puts raw production inputs and outputs in front of a human, which is a data-handling decision, not a UI feature. If traffic can contain personal or regulated data, who can open the queue and what is masked before it gets there is a question you must have an answer to before you turn one on. ## The shape of a good answer Say what it is (structured human review), name the three entry routes, say the output is human-sourced feedback on the run, and finish with the two things that separate people who have run one: the queue is how corrections become dataset examples, and its throughput is bounded by reviewer hours rather than by the sampling rate you type in.