Caption-and-index, page-image embeddings, or late interaction — how do you pick for a figure-heavy slide corpus?

## The problem this decision exists to solve Take a medical-affairs team with 40,000 conference slides. A typical answer is a Kaplan-Meier survival curve with a hand-drawn annotation, sitting on a slide whose only extractable text is a title and a footnote. A text-only pipeline indexes the title and the footnote, so the page is effectively invisible to any query about the shape of the curve. Multimodal retrieval exists because the information is *in the pixels*, and every architecture below is a different answer to "how much of the page do we keep, and at what price?" ## Design 1 — caption-and-index Render each page, ask a vision-language model to describe it, store the description, embed the description with an ordinary text model. Retrieval is then plain text retrieval and every tool you already own keeps working: keyword hybrid, rerankers, filters, existing dashboards. The fatal property is that captioning is **lossy compression performed before the query is known**. A caption that says "survival curve comparing two arms" cannot answer "which slide shows the crossover at month 18?" Adding detail helps only until the caption itself becomes long, expensive to generate, and diluted when embedded into a single vector. Ingest cost is also real: one VLM call per page, times 40,000, times every re-caption when you change models. It is still the right choice when pages are text-dominant, when you need the caption text for other purposes anyway, or when your retrieval stack cannot store image-derived vectors at all. ## Design 2 — unified single-vector page embeddings A newer class of embedding models ingests the page image directly (optionally with any text) and emits **one** vector in a space shared with text queries. Index size per page is identical to a text chunk, latency is one nearest-neighbour lookup, and there is no captioning step to go stale. Because the model saw the pixels, layout and figure content influence the vector in ways a caption never captures. The limit is representational: one vector must summarise a whole dense page. On a slide with four unrelated panels, the single vector is a blend, and a query targeting one panel competes with the other three. Some of these models expose Matryoshka-style truncatable dimensions, so you can trade a little accuracy for a smaller index without changing models. ## Design 3 — late interaction over page images ColPali-style models keep the per-patch vectors the vision encoder already produced — on the order of a thousand small vectors per page — and score a query by summing, over each query token, its best match against any patch (MaxSim). No OCR anywhere in the pipeline. This preserves fine-grained, localised evidence, which is exactly why it dominates on figure-heavy documents. The bill is the point of the question. A rough arithmetic: 40,000 pages × ~1,000 patch vectors × 128 dims × 2 bytes ≈ 10 GB of raw vectors, against ~0.25 GB for one 1,536-dim float vector per page. Scoring is also heavier — you normally run an approximate first stage to fetch candidates, then MaxSim-rerank only those, which adds a stage to operate and tune. ## The pooling compromise The two extremes are not the only options. Patch vectors can be **pooled** — clustered or averaged down from ~1,000 to a few dozen per page — which recovers most of the storage while keeping much of the localisation benefit. Dimension reduction and binary or scalar quantisation cut it further. In practice the interesting engineering is here, not in the binary choice: pooled multi-vector plus quantisation often lands within a point or two of full late interaction at a fraction of the index. ## How to actually decide Build a small labelled set from *your* corpus and your users' real questions, then measure recall at the k you can afford to put in the generator's context. Report cost alongside quality: index GB, ingest cost per 10,000 pages, and p95 query latency. Segment the results — if late interaction only wins on the 20% of queries that are chart-shaped, route by query type or index only the figure-heavy subset that way. Be honest that this is contested ground in 2026: the models on both sides are improving fast, and the right answer for a 5,000-page corpus (take the recall, the storage is free) is not the right answer for 50 million pages. ## Failure signals Captioning is failing when users ask about details that never appear in your captions. Single-vector is failing when the correct page is retrieved at rank 30 rather than rank 3 on dense multi-panel slides. Late interaction is failing operationally, not qualitatively — index growth, rebuild time and rerank latency are what force the retreat.