What is a Langfuse annotation queue, and how does human review become scores?

## What a queue is Human review is the only source of labels that does not itself need validating, and it is the scarcest. A Langfuse **annotation queue** is the machinery for spending it well: a named work list of trace or observation items, backed by a defined set of score configs, that reviewers work through in the UI. Creating one involves two choices. Which **score configs** may be applied — that is, what questions the reviewer is being asked, with their data types and allowed values fixed in advance. And what **enters** the queue: items can be added ad hoc from a trace you are looking at, or in bulk from a filtered view of traffic, so a queue can be 'everything a user thumbs-downed this week' or 'a random sample of the new prompt's output'. ## The review loop A reviewer opening a queue item sees the execution itself, not a spreadsheet row: the input, the final output, the nested observations, retrieved context, tool calls. They record their verdict through the controls the score configs define — a bounded slider for a numeric score, a fixed label list for a categorical one — and can leave a comment explaining it. Constraining the input is the point: free-text labelling is where inter-reviewer vocabularies diverge, and no amount of downstream cleaning fixes it. What lands is an ordinary score on the same trace, with source `ANNOTATION` — the same object shape your SDK writes with source `API` and an evaluator writes with source `EVAL`. Every dashboard, filter and drill-down that works for one works for all three. ## Why the shared shape matters Three workflows fall out of it. **Judge checking.** Route the same traces through an LLM judge and a human queue, then compare the two score series on identical traces. Where they disagree, you have concrete cases to look at. (How you decide whether the judge is *good enough* from that comparison — agreement statistics, calibration — is evaluation methodology, not a Langfuse feature.) **Ground-truth harvesting.** A reviewed trace that is confirmed wrong is exactly the input a regression set wants. Create a dataset item whose `source_trace_id` is that trace, and the failure becomes a permanent test case that keeps its provenance. **Triage.** Queues give a bad-output investigation a shape: a filter defines the population, the queue holds the work, and the resulting scores say how much of it was actually bad, rather than the loudest three examples anyone happened to see. ## Operating a queue The hard constraint is human throughput. A reviewer handles on the order of tens of traces an hour, which is nothing against production volume, so the real design question is *what gets queued*. Uniform random sampling gives you an unbiased quality estimate but wastes most of the effort on unremarkable traces; targeting — thumbs-down traces, high-cost traces, a specific tool's failures, a new release's first hour — finds problems faster but tells you nothing about the base rate. Most teams run both: a small unbiased stream for the trend, and targeted queues for investigations. Two more practicalities. Traces contain prompt and completion content, so who may open a queue is a data-access question, not just a workflow one; in regulated settings that pushes toward self-hosting and masking. And queue items are only as good as the trace behind them — if retrieved context is not recorded as an observation, a reviewer cannot tell a retrieval failure from a generation failure, and will label the wrong thing.