Why do agents load context through tools on demand instead of preloading it?

## The two shapes of getting context in An LLM only sees what is in its context window. There are exactly two ways to get something in there: put it in before the model runs (**preloading**), or give the model a tool it can call to pull it in mid-run (**just-in-time**, or JIT, loading). Everything else — retrieval quality, caching, compaction — sits on top of that choice. Preloading is the older habit, inherited from single-shot prompting: assemble every document, schema, or API page the task might touch, concatenate it, send it. JIT loading instead puts *references* in the prompt — a directory listing, a set of record IDs, a table catalog, a search endpoint — and lets the model decide which of them to expand into full content. ## Why preloading stops working Three pressures push against preloading: **Attention is a budget, not free space.** Every token in the window competes for the model's attention. A useful framing is that the goal is the *smallest high-signal token set*, not the largest one that fits. When 95% of the prompt is irrelevant, the relevant 5% is harder to find and easier to contradict. **Effective context is smaller than advertised.** Frontier models ship million-token windows, but long-context benchmark families agree that usable quality holds over roughly 40–70% of the advertised length, and that it falls off a cliff rather than degrading smoothly. This degradation is commonly called *context rot*. A prompt that technically fits can still produce worse answers than a much shorter one. **You often cannot know in advance.** For a real agent task — investigate this incident, answer this question against the warehouse — the set of documents that matter is discovered *during* the work. Preloading forces you to guess the superset, and the superset is usually enormous. ## What JIT loading actually looks like The agent's starting prompt contains identifiers and enough metadata to choose between them. A file agent gets a directory listing. A database agent gets a table catalog with names and row counts. A support agent gets ticket IDs with subjects and dates. Then a tool — read this path, describe this table, open this ticket — returns the full payload, which is appended to the conversation as a tool result. Two properties matter. First, the model does the selection, so selection benefits from everything the model knows about the task, not just from a similarity score computed before the task started. Second, the content that lands in the window has already been *justified* by a decision — it is there because the model asked for it. ## What it costs JIT loading is not free, and a good interview answer names the costs: - **Round trips.** Each fetch is another inference call. A three-hop lookup is three times the time-to-answer of a preloaded prompt, and the hops are serial when later choices depend on earlier results. - **Wrong fetches.** The model can pick the wrong file, or open five when one would do. Every wrong fetch costs both a round trip and window space. - **Non-determinism.** Which context an answer used now varies per run, which complicates reproducing a bad answer and evaluating the system. - **Cache pressure.** A prompt that changes shape mid-session interacts badly with prefix caching unless the loaded content is appended behind a stable prefix. ## Choosing between them The honest rule is about *hit rate and size*. Preload material that is small, stable, and needed on nearly every turn — a short style guide, the two schemas the agent always joins, a company glossary. Load just in time when the candidate set is large, the needed subset is small, and which subset is needed depends on the task. Hybrids are normal and usually correct: preload a compact index or catalog so the model can choose without a round trip, and JIT-load the bodies. That is the same structure as progressive disclosure — identifiers and metadata up front, content on request. ## How you would know it is working Instrument it. Track how many fetches a task takes, how many of the fetched items were actually cited or used, and the ratio of loaded tokens to task-relevant tokens. A rising fetch count with a flat success rate means the agent is groping; a low use-rate on fetched content means the metadata it selects from is too weak to discriminate. Both are fixable, and neither is visible if you only look at the final answer.