When does filling Gemini's million-token window beat retrieval over the same corpus?

## Reframing the question With a million-token window the interesting question stops being "can I fit it" and becomes "should I pay to fit it, every single call". The window is not storage. It is a per-request allowance that is billed and prefilled afresh on each turn unless caching intervenes. ## The three forces **1. Does it fit — and will it keep fitting?** A million tokens is roughly a few thousand pages of prose. That covers a contract set, a mid-sized codebase, a quarter of support transcripts. It does not cover a company wiki, a document store with millions of files, or anything that grows without bound. Even when today's corpus fits, an architecture that assumes it always will is fragile; retrieval degrades gracefully as data grows, full-context loading falls off a cliff. **2. What does each call cost, and how long does it take?** Input tokens are charged per request. A 700k-token prompt asked ten times costs seven million input tokens unless cached. On top of that, Gemini 2.5 Pro prices prompts above roughly 200k tokens at an elevated per-token rate (as of mid-2026), so the largest prompts are not merely proportionally more expensive but disproportionately so. Latency follows the same curve: prefill work grows with prompt length, and time to first token on a very long prompt is visibly worse than on a retrieved 8k-token prompt. Context caching changes this arithmetic but does not reverse it. A cached prefix is discounted and skips re-prefill, which is what makes a fifteen-question session over one document economical. It still counts toward the prompt size, still occupies the long-prompt tier, and still costs storage while it lives. Caching turns "expensive on every call" into "expensive once, cheap thereafter" — which is precisely why reuse count is the deciding variable. **3. What is the task actually asking?** This is where full-context genuinely outclasses retrieval. Chunk retrieval assumes the answer lives in a few passages that a query embedding can find. That assumption breaks for: - **Aggregation** — "how many of these 400 tickets mention the same root cause" needs all of them, not the top-k most similar. - **Cross-reference chains** — a definition in clause 2 constraining an obligation in clause 87; retrieval surfaces one and misses the link. - **Global structure** — "summarise the argument of this book", "how does auth flow through this service", where meaning is distributed rather than located. - **Poorly-phrased queries** — when the user's words do not lexically or semantically resemble the passage they need, retrieval simply fails to surface it while a full-context model can still find it. Conversely, retrieval remains superior when the corpus dwarfs the window, when documents carry per-user access control that must be enforced at selection time, when content changes constantly and you need this minute's version, and when you serve high query volume with no reuse — thousands of one-shot questions across thousands of distinct documents, where every request would otherwise pay a giant prefill. One more honest caveat: a model that *can* attend to a million tokens does not attend to all of them equally well. Simple needle-retrieval over long inputs is strong, but multi-hop reasoning and exhaustive aggregation get harder as inputs grow. Long context reduces retrieval engineering; it does not make context length free of quality effects. ## The decision, in practice Ask in order: (a) Does the material fit with headroom, today and next year? (b) How many questions will be asked against the same material in one session — one, or twenty? (c) Is the task pinpoint lookup, or whole-document reasoning? (d) Is there access control or freshness pressure on individual documents? Fit + high reuse + whole-document reasoning ⇒ load it all, and put the stable part in a cache with a TTL matched to the session. Doesn't fit, or one-shot queries at volume, or per-document ACLs ⇒ retrieve. ## The hybrid most systems land on Mature systems rarely pick a pure strategy. They retrieve *coarsely* — whole documents rather than 500-token chunks, recall-oriented rather than precision-oriented, because the window can absorb the slack — assemble a few hundred thousand tokens of plausibly-relevant material, and let the model do the fine-grained selection that a chunker used to do badly. The retrieved-and-stable portion goes into a cache for the session. This keeps the corpus unbounded while capturing most of the reasoning benefit of a long window, and it is the answer that reads as production experience rather than enthusiasm for a spec sheet.