How would you prove page-image retrieval beats an OCR-then-embed baseline on figure-heavy decks?

## Why this comparison needs discipline Both pipelines will retrieve *something* for every query, and both will look plausible in a demo. The difference lives in the tail: queries whose answer sits in a figure that produced no extractable text. If your evaluation set is drawn from whatever questions were easy to write, it will over-sample text-answerable queries and the two pipelines will look equivalent. The evaluation design is therefore the whole exercise. ## The two pipelines, held constant The **baseline**: render or parse each page, extract its text, embed the text, index it. The **candidate**: render each page as an image and index it with a page-image embedding model or a late-interaction model. Everything downstream must be identical — same query set, same k, same tie-breaking, same hardware for latency numbers. If you change the generator model at the same time you change the retriever, you have measured nothing. ## Metrics Retrieval is a ranking problem, so use ranking metrics. **nDCG@k** rewards putting the right page high and is the standard headline for visual-document retrieval; ViDoRe reports nDCG@5. **Recall@k** answers the question that actually matters downstream: is the answering page inside the k pages I can afford to put into the generator's context? If your budget allows five page images, recall@5 is the number that decides whether the system can possibly answer. **MRR** is fine when exactly one page is correct. Report quality *next to* cost: index bytes per 1,000 pages, ingest cost, and p95 query latency. A pipeline that wins nDCG by four points and costs fifty times the index is a tradeoff to argue, not a result to celebrate. ## The public benchmark and its limits ViDoRe (the Visual Document Retrieval benchmark) exists precisely to score retrieval over page images against text-extraction baselines across figure-heavy, table-heavy and multilingual documents, and it is what the ColPali family and the unified page-embedding models report on. Use it to shortlist models — it will tell you quickly whether a candidate is in the right class. Do not use it to decide. Public benchmarks suffer contamination and saturation, their query distribution is not yours, and leaderboard position is increasingly a fine distinction between models that behave differently on real corpora. In 2026 the top of ViDoRe is crowded; the gaps that matter are on your data. ## Building the in-house set Mine real questions — support tickets, search logs, the questions the medical-affairs team actually asks — rather than inventing them, because invented queries drift toward the vocabulary that appears in the text. Aim for a few hundred queries; that is usually enough to separate pipelines that differ meaningfully, and small enough that a human can label honestly. Label by identifying the page(s) that genuinely answer each query. Have a second annotator label an overlapping subset and check agreement — if two people disagree about which page answers a question, the metric is measuring annotation noise. ## Slicing is where the answer appears Tag every query or page with a type: text-dense prose, dense multi-panel slide, chart-only, table-heavy, scanned or handwritten, non-English. Then report per slice. The expected pattern: near-parity on text-dense pages, where a good text pipeline is genuinely competitive and much cheaper; a large gap on chart-only and handwritten pages, where extraction returns a title and nothing else. A single aggregate number averages those together and can easily show a modest overall win that hides a decisive one on the slice you care about. It also tells you the cheapest architecture: route by page type and pay for the expensive index only where it earns its keep. ## Diagnosing rather than scoring Beyond metrics, read the failures. For each query the baseline missed, look at what its extraction actually produced for the correct page — usually a title, a footnote and nothing else, which is the concrete artifact that convinces a sceptical stakeholder far better than a decimal point. For each query the image pipeline missed, check whether the detail was simply too small for the encoder's patch resolution, which is a ceiling no scoring change will lift. ## Guarding the result Fix the query set and version it, so later model swaps are compared against the same yardstick. Re-check periodically for drift as the corpus grows and users' questions change. And state your assumptions in the write-up — extraction quality, k, hardware — because the most common way this comparison goes wrong is that someone tuned one side and not the other.