How do you stop just-in-time loading from invalidating a cached prompt prefix?

## What prefix caching gives you and what it demands Providers cache the computed state of a prompt prefix so that a follow-up request sharing that prefix skips recomputing it. The reported savings are large — commonly cited around 90% on cost and 85% on latency for the cached portion — which makes cache hit rate a real budget line for any multi-turn agent. The demand is exact-prefix matching. The cache is keyed on the token sequence from the start of the prompt. It survives up to the first position where this request differs from the cached one; from there on, everything is recomputed. That single property drives every rule below. ## Why JIT loading is the natural enemy of a stable prefix Just-in-time loading means the prompt grows unpredictably during a session: a document arrives at turn three, a schema at turn five. If the harness places that content anywhere except the end, it moves the boundary. Three common ways teams break their own cache: - **Injecting fetched content into the system block** — a tempting way to make it feel authoritative, and the worst possible position, because it invalidates literally everything. - **Re-sorting or re-grouping** — collecting all loaded documents into one 'context' section, re-sorted by relevance each turn, so the section's bytes change every turn. - **Editing earlier turns** — trimming old tool results, replacing a payload with a summary, or renumbering references. Each edit resets the cache from that point forward. A fourth, subtler one: **non-deterministic serialization**. If the tool result is a JSON object whose key order or whitespace varies between runs, or carries a timestamp, the bytes differ even when the content does not, and the prefix stops matching for reasons nobody can see in the rendered prompt. ## The discipline: stable prefix, append-only tail Structure the request in layers ordered by volatility, most stable first: 1. **System instructions** — fixed for the deployment. 2. **Tool definitions** — fixed per session. 3. **Preloaded catalog or index** — the tier-one listing, if it is stable for the session. 4. **Conversation history** — grows at the end. 5. **Tool results from JIT fetches** — appended in arrival order, never reordered. With this ordering, every fetch extends the prompt rather than mutating it, so each turn's cache boundary moves *forward* and the previously cached portion stays valid. This is the concrete meaning of the advice to design context append-only. Two corollaries. First, anything volatile — the current timestamp, a per-request user ID, a randomized instruction — must go *after* the stable layers, not in the system block, or it poisons the whole prefix on every request. Second, a catalog that refreshes mid-session should be treated as volatile and placed with the conversation, not with the instructions. ## When breaking the cache is the right call The discipline is not absolute. Window space is finite, and eventually an agent must reclaim it — by dropping stale tool results, summarizing, or reinitializing the session. Each of those invalidates the cache from the edit point, and that is sometimes correct: a window filled with dead payloads produces worse answers, and worse answers cost more than a cold prefix. The way to reason about it is to compare the recurring cost of carrying the dead tokens — paid every turn for the rest of the session — against the one-time cost of reprocessing the prefix. Cleanups that are rare, batched, and large win; cleanups that are frequent and small lose, because you pay a cold prefix repeatedly to reclaim a little space each time. Batching is the lever: do one substantial cleanup rather than trimming continuously. ## What to measure Instrument cache hit rate per turn alongside token counts. A hit rate that collapses at a particular turn number points at a specific mutation — often a harness feature nobody associated with the cache, like re-rendering a header with a timestamp. Also log the *position* of the first mismatch when you can, because that identifies the offending layer directly. ## The interview-shaped summary JIT loading and prefix caching are compatible, but only under an ordering rule: stable things first, never edited; fetched things appended at the end. Where they genuinely conflict — reclaiming space from a full window — treat cache invalidation as a cost to be paid deliberately and in batches, not accidentally on every turn.