Which deploy-time changes silently invalidate every cached prompt prefix?

## What the lookup is keyed on A prompt cache entry is not addressed by a name you choose. It is addressed by content: the exact sequence of tokens making up the prefix, together with the identity of the model that produced the stored state. Change either side of that pair and the lookup finds nothing. There is no partial credit, no migration, and no warning — the request simply behaves as if it were the first one you ever sent. ## The invalidators you control **Model version.** This is the one that surprises teams. Upgrading to a newer snapshot of the same model family, with a prompt that has not changed by a single character, invalidates everything. Internal state is model-specific and cannot be reused across versions. Pinning a model version is therefore also a caching decision: a floating pin means an invisible, vendor-scheduled invalidation. **The stable prefix content.** Any edit to the material you deliberately placed in the cached region — system prompt wording, few-shot examples, a policy document, formatting instructions — invalidates from the edit point onward. This includes changes nobody thought of as prompt changes: a build SHA or deploy timestamp templated into the header, a feature flag rendered as a line of text, a locale string. **Tool definitions.** Tool schemas usually sit in the cached region ahead of the conversation, so the tool list is part of the key. Adding a tool, removing one, reordering the list, or rewording a single description all invalidate. Teams that ship tools independently of prompts are effectively shipping cache invalidations they never planned. ## The ones you do not control Providers may retire a model snapshot, change the default when you have not pinned one, or route your traffic to different serving capacity. Some implementations key additional request-envelope settings into the lookup beyond the raw prompt text. The honest posture is: enumerate what you know invalidates, then verify empirically from the write-versus-read token split rather than assuming any change is safe. ## What a fleet-wide invalidation feels like The failure mode is not subtle, but it is easy to misattribute. At the moment of deploy every in-flight and subsequent request misses simultaneously. You see a step change in input cost, a jump in time-to-first-token across the board, and — if you are near a rate or concurrency limit — a burst of contention as every worker reprocesses the same large prefix at once. Because it coincides with a release, it is frequently blamed on the new prompt's quality or on the model upgrade being 'slower', when it is purely the cold-start cost of the first write per prefix. The damage scales with prefix size. A deploy that invalidates a 500-token system prompt is noise; one that invalidates a 120k-token document prefix used by every request is a real incident on the cost graph. ## Absorbing it **Treat prompt, tool definitions and model pin as one versioned artifact.** If they ship together, you always know when a cache-invalidating change lands, and you can annotate the cost graph. **Roll gradually.** A canary or staged rollout spreads the write burst over time instead of concentrating it, and keeps the old prefix serving warm reads for the untouched share of traffic. **Pre-warm deliberately.** Before shifting traffic, send one representative request per distinct prefix variant so the entry exists when real users arrive. This is a small, bounded number of writes you would have paid anyway — just moved off the user-facing path. **Keep variant count low.** Every distinct stable prefix is its own entry with its own warm-up cost. Per-tenant or per-locale prefix variants multiply the deploy penalty linearly, which is an argument for hoisting genuinely shared material above anything that varies. **Do not chase micro-edits.** Once you understand the mechanics, the temptation is to freeze the prompt forever. That is the wrong lesson. Prompt quality is worth far more than a cache write; the point is to know the cost, batch changes into deliberate releases rather than dribbling them out, and stop letting incidental content like timestamps invalidate entries for no benefit at all.