Preload 40k tokens of API reference, or take three round trips to fetch one endpoint?

## Why round trips are the right unit The instinct is to compare token cost in dollars, and that comparison usually says 'fetch on demand' loudly — 40k tokens per turn adds up. But token price is rarely what decides this in production. Two other things do: **answer quality**, which favours the small clean window, and **wall-clock latency**, which favours the fat static prompt. Latency is measured in model round trips, and a round trip is expensive in a way tokens are not: it is a full inference call, including think time. So count hops. A preloaded prompt answers in one call. A JIT agent that must list, then read the index, then open the endpoint page, then answer, is four calls. If each call is two to four seconds, the JIT path adds six to twelve seconds before the first useful token reaches the user. ## The variables that actually decide it **Hit rate.** How often is the same material needed? If nearly every task touches the same three endpoints, preloading them costs 3k tokens and saves a hop on every request. If the needed endpoint is unpredictable across 200 candidates, preloading all 200 to serve one is a terrible bargain. **Hop depth and seriality.** JIT costs are not one extra call; they are however many *dependent* calls discovery takes. Serial hops are the killer, because each one blocks on the previous result. If the agent can fire several fetches in one turn — it already knows three identifiers — the cost collapses to roughly one hop regardless of how many items are read. **Who is waiting.** An interactive assistant with a human watching has a latency budget measured in seconds; a batch pipeline processing overnight has none. The same architecture is wrong in one setting and right in the other. **Window pressure.** If the task is long-horizon, the 40k preloaded tokens are not a one-time cost — they persist through every turn, competing with the accumulating work product and pulling the session toward the degradation zone sooner. **Determinism.** Preloaded context is identical every run, which makes failures reproducible and evals stable. JIT context varies, so the same question can succeed and fail on different runs for reasons invisible in the final output. ## The hybrid that usually wins For an API reference specifically: preload a compact index — endpoint names, one-line purposes, maybe method and path shape — at a couple of thousand tokens, and fetch the full page for the one or two endpoints the task uses. That buys a single extra round trip instead of three, because the index removes the discovery hops, and it keeps roughly 95% of the token saving. The pattern generalizes: **pay tokens for the map, pay round trips for the territory.** A second lever is batching the fetch. If the index makes it obvious that two endpoints are needed, the agent should request both in the same turn rather than serializing. Whether the harness supports that is an implementation detail, but designing the tool surface so multiple identifiers can be requested at once is a deliberate choice that directly cuts hop count. ## Latency mitigations that do not change the architecture - **Speculative prefetch**: while the model is reasoning, the harness fetches the two most likely items so the tool result is already warm. Cheap in tokens only if you discard what is unused. - **Streaming**: it does not reduce hops, but it changes the perceived cost — the user sees progress during the middle hops rather than a blank wait. - **Warm caching of the preloaded part**: if the index is stable, it can sit behind a cacheable prefix, so the tokens cost near-nothing after the first turn and the argument shifts further toward preloading the index. ## What to measure before deciding Do not argue this from first principles when you can instrument it. Log, per task: number of fetches, which items were fetched, which were actually used in the answer, total tokens, and end-to-end latency split into model time and tool time. Two numbers settle most arguments — the **hit rate** of each preload candidate, and the **hop distribution** of the JIT path. If the hit rate is high and the hop count is above two, preload. If the hit rate is low, keep the fetch and work on cutting hops with a better index. ## The answer an interviewer wants Not 'JIT is better'. They want you to say it depends on hit rate, hop depth and who is waiting; to propose the index-plus-fetch hybrid; to name the measurement that decides it; and to be honest that preloading still wins for small, stable, high-hit-rate material under a tight latency budget.