How do you set the confidence threshold that sends an extracted field to human review?

## The threshold is an operating point, not a constant Every extraction pipeline that touches money or goods ends up with a review tier. The engineering question is never "is the model confident" in the abstract; it is where to put the cut between auto-post and human review, and that is a choice on a curve you have to measure. The curve has two axes. As the threshold rises, more fields route to review — that is labour cost and latency. As it falls, more fields auto-post — and the error rate among auto-posted fields rises, because you are admitting progressively weaker evidence. You cannot optimise both. What you can do is quantify the exchange rate and let the business pick the point. ## Building the sample you tune against The measurement needs ground truth: a labelled set drawn from real traffic, not from the clean documents someone had handy. Stratify it by the dimensions that actually move accuracy — document class, source channel (born-digital versus fax versus phone photo), vendor template, and language. A few hundred documents per meaningful stratum is usually enough to place a threshold; the strata matter more than the total. Then sweep. For each cutoff, compute review rate and auto-post error rate per field. The result is often surprising and always useful: some fields are safe to auto-post at almost any threshold, while one or two carry nearly all the residual error and would need a cutoff so high that nothing auto-posts. Those fields should simply be routed to review unconditionally, or dropped from automation until the upstream capture improves. ## Per-field thresholds, because the cost of error is not uniform A single global threshold is the most common mistake. On an arrival notice, getting a shipper's address slightly wrong is an annoyance someone fixes later; getting a container number wrong misroutes a physical box, and getting a dangerous-goods checkbox wrong is a regulatory event. The threshold should reflect the cost of a wrong value passing, which means per-field, sometimes per-field-per-document-class. The same logic applies to routing granularity. Sometimes one weak field should send only that field for review, with the rest posting; sometimes a weak field on a legally-signed document means the whole document is held. Both are legitimate; decide deliberately rather than inheriting whatever the framework does. ## Confidence should be a composite Relying on a single model-emitted score is fragile, and on the model tiers that read messy documents best, that score often does not exist. Build a composite from independent signals: - **Recognition confidence** from a classical OCR pass, where you run one. It is genuinely informative about smudged, skewed, or low-contrast glyphs. - **Validations from the domain.** These are the strongest signals available because they do not depend on the model at all: a shipping-container identifier carries a check digit, line-item amounts must sum to the stated total, a date must fall inside a plausible window relative to the sailing date, an enum value must be in the enum. A failed validation should override a high confidence score outright. - **Grounding checks.** A field whose value cannot be located in the page's recognised text, or whose box falls on whitespace, is a hallucination candidate regardless of what score accompanied it. - **Agreement between extractors.** Running a second, cheaper model — or the same model on a re-rendered page — and comparing gives a disagreement signal that correlates well with error. It costs a second pass, so it is usually reserved for high-stakes fields. A note on calibration: raw model scores are rarely calibrated probabilities, so do not read 0.9 as "90% likely correct". What you need is that the score *ranks* correctness usefully; the mapping from score to actual error rate comes from your labelled sweep, not from the number's face value. ## Keeping it honest in production A threshold tuned once decays. Input quality shifts when a customer swaps scanners, a carrier redesigns a template, or a model version changes. Two practices keep it honest. First, **sample the auto-posted stream**. Pull a small random slice of documents that were never reviewed, have a human check them, and track the measured error rate over time. Without this you are blind to exactly the population you care about — reviewed documents tell you nothing about what slipped through. Second, **capture reviewer corrections as labels**. Every correction is a labelled example, and it costs nothing extra to store. Over months that stream becomes your refreshed evaluation set, your drift detector, and — if you go that way — training data. Beware its bias though: corrections come only from documents that were routed to review, so this stream over-represents hard cases and cannot replace the random audit. The outcome to aim for is a stated, measured operating point per field: "printed container numbers auto-post with a measured residual error of 0.3%, handwritten weights and the hazardous checkbox always go to review, and 11% of documents touch a human." That is an answer an operations lead can act on, which is what the question is really asking for.