In document extraction, why demand a bounding box and page number per extracted field?

## The output contract, not just the output An extraction pipeline that returns free-form prose about a document is not an extraction pipeline. What downstream systems need is a record: named fields, typed values, and enough evidence to defend each one. That contract has two halves. The **schema** fixes what may appear: field names, types (a date is a date, a weight is a number with a unit), enumerations for closed sets like incoterms or container types, and — critically — an explicit representation of absence. A pipeline without an explicit null teaches the model that every field must be filled, which is precisely how you get a plausible consignee on a document that never named one. "Return null when the document does not state it" is one of the highest-value instructions in the whole system. The **provenance** half attaches, to every field, where the value came from: page number, a bounding box in page coordinates, the literal text spanned by that box, and whatever confidence signal the tier provides. ## What grounding buys **Review throughput.** This is the immediate operational win. A clerk verifying a bill of lading with twenty fields across seven scanned pages spends most of their time *finding* the value, not judging it. With a box, the UI highlights the region and the clerk answers yes or no in a second. On a queue of thousands of documents a day, that difference decides whether human review is affordable at all. **Hallucination detection.** The model tiers that read messy documents best are also the ones that fail most plausibly: asked for a container number on a fax-degraded page, a model may emit a well-formed container number that appears nowhere. Grounding gives you cheap, automatic checks. Does the box fall on ink or on whitespace? Does the text inside the box match the value that was returned, allowing for normalisation? Does the box sit on the page the field is supposed to be on? A field that fails these is quarantined without a human ever looking at it. **Audit and dispute.** In freight, customs and finance, someone will eventually ask why your system posted a particular weight. "The model said so" is not an answer. "Page 3, this rectangle, this text, extracted at this time by this model version" is. That record also makes regressions tractable: when accuracy drops after a model change, you can diff not just the values but where they were read from. **Evaluation.** Boxes let you score *reading the right thing* separately from *reading it correctly*. A pipeline that gets the right value from the wrong region is one layout change away from silently breaking, and only provenance exposes that. ## How to get boxes you can trust Here the practical detail matters. General multimodal models are good at reading text and unreliable at emitting exact pixel coordinates — coordinates drift, or arrive in a normalised space you have to guess at. Purpose-built document models in the OCR-VLM tier are far better, because grounded layout output is what they were trained to produce. The most robust pattern does not ask the model for geometry at all. Run a pass that yields word-level boxes — a classical OCR engine, or a document model that emits blocks with coordinates — and keep them. Then take each extracted field value and match it back against those words to derive its box. String matching with normalisation (whitespace, case, punctuation, thousands separators) resolves most fields; fuzzy matching handles the rest. The match itself becomes a signal: a field whose value cannot be located anywhere in the page's recognised text is a hallucination candidate, flagged automatically. ## Schema design choices that pay off Keep the schema flat and shallow where the domain allows; deeply nested structures raise both the error rate and the difficulty of reviewing a single field. Represent repeated structures — line items — as arrays of objects, each item carrying its own provenance, so a reviewer can approve nineteen rows and correct one. Normalise at the boundary, but store the raw string too: "12,500.00 KG" is what the page said, 12500.0 with unit KG is what the system needs, and disagreements between them are a useful alarm. Add cross-field invariants where the domain provides them — a total that must equal the sum of line items, an identifier with a check digit — because a validation failure is a far stronger signal than any model-reported confidence. ## What it costs Grounding is not free. Emitting boxes and source spans adds output size and, in the matching approach, a second pass to maintain. The honest framing is that you pay a modest engineering and runtime cost to convert an unverifiable output into a reviewable one — and on documents that move money or goods, an unverifiable output was never shippable in the first place.