Besides TTL expiry, what can drop a cached prompt prefix before its window ends?

## Best-effort by design A prompt cache is a performance optimisation layered onto inference infrastructure, not a durable store with a contract. Providers state this plainly: an entry *typically* survives its TTL, and the TTL bounds how long it *may* live — it does not guarantee how long it *will*. Reasoning about it as a guarantee is the root of most production surprises here. ## Ways an entry disappears early **Capacity pressure.** The stored state consumes real memory on serving hardware. Under load, or during maintenance and rebalancing, entries can be reclaimed before their window expires. You get no notification; the next request is simply a write. **Routing.** Your request has to reach capacity that holds the entry. Fleet changes, deployments on the provider side, regional failover, or a differently routed request can all land you somewhere that has never seen your prefix. This is invisible from the outside and shows up only as an unexplained write in your usage accounting. **Content divergence.** Not eviction strictly, but the same practical outcome: if anything ahead of or inside the cached region differs from what was stored — a changed model version, an edited instruction, a reordered tool list — the lookup misses even though a perfectly good entry still exists for the old content. ## Scope: caches are private Entries are keyed to the account or organisation that created them. Two unrelated customers sending byte-identical system prompts do not share anything, and popular public prompts are not warmed globally on your behalf. This is a privacy and isolation property first — a shared cache would leak information about what other customers send, and timing differences alone could reveal it — and only incidentally a performance limitation. Within a single organisation, separate keys, projects or regional deployments may also behave as distinct cache domains, which is worth verifying rather than assuming when you fan traffic across several of them. A practical corollary: if you shard the same workload across multiple keys or regions for rate-limit headroom, you have multiplied the number of prefixes that must be independently warmed. That can quietly halve an expected hit rate. ## Designing for the miss Because a miss is always possible, the miss path is the path you must engineer for. **Timeouts.** Size client and gateway timeouts against uncached processing of the full prefix. A timeout tuned to warm-cache first-token latency will fire spuriously the first time an entry is gone, converting a cost event into an availability event — and the retry then pays the write again. **Capacity.** If a burst of traffic arrives cold — after a deploy, a failover, or an eviction — every request reprocesses the same large prefix concurrently. Bound concurrency so a cold start degrades throughput gracefully instead of saturating your rate limits. **SLOs.** Publish latency targets your system can meet without a hit, and treat the hit as headroom. Teams that set a p95 from warm-cache measurements discover on the first eviction that the target was never achievable. **Correctness.** Nothing about the answer changes on a miss — the full prompt is still sent and processed — so caching must never be load-bearing for behaviour. If some content only reaches the model 'because it is cached', the design is already wrong. ## How you would know The only observable is the per-response usage split between cache-write and cache-read tokens. A shared, stable prefix under steady traffic should show writes as a low, roughly flat count with reads scaling with volume. Sporadic unexplained writes that do not line up with deploys or with gaps in traffic are the fingerprint of early eviction or routing changes — worth accepting as background noise rather than chasing, provided the rate stays low. If it does not, the honest conclusion is usually that the workload's shape, not the provider, is the problem.