When does enabling prompt caching make a workload more expensive?

## The failure condition in one line Caching costs money up front and refunds it on reuse. So the loss condition is simply: **a write that is never read.** Everything below is a way of arriving at that state. ## Case 1 — the genuine one-shot A document-processing service summarizes each uploaded file exactly once. A user uploads an 8k-token contract; the service sends it with a short instruction and gets a summary back. If caching is switched on for that call, those 8k tokens are billed as a cache write at roughly 1.25x base instead of 1x — about a **25% surcharge** on the input for that request — and nothing ever reads the entry, because the next upload is a different contract. The latency picture is equally unhelpful. The first request must read the prompt regardless; caching only helps a *subsequent* request that reuses it. So a one-shot call gets neither benefit and pays the premium. This is the cleanest example of caching being straightforwardly wrong, and it is common in practice because caching often gets enabled globally in a shared client wrapper rather than per request class. ## Case 2 — a "stable" prefix that is not stable More insidious: the prefix looks fixed but carries something that varies. A tenant name, an account tier, a timestamp, a user's display name, a freshly shuffled example set — any of these embedded in the block makes every request a distinct prefix. From the billing system's perspective this is a firehose of writes with essentially no reads. This is worth flagging precisely because it *looks* like it should be working; the code says caching is on, and the bill goes up rather than down. The diagnosis lives in the reported token counts (below); the correctness question of what belongs in the stable block belongs to prompt structure, not to this cost analysis. ## Case 3 — traffic slower than the entry lives Even with a perfectly stable prefix and genuine repetition, timing can kill it. Cache entries have a finite lifetime measured in minutes for the common tier. A service that touches a given prefix once an hour finds no live entry every time: write, expire unused, write again. Cost settles at the write multiplier — around 25% above baseline forever. The subtlety is that **aggregate volume does not save you**; per-prefix arrival rate does. A system doing a million calls a day spread across 50,000 distinct customer-specific prefixes may have almost no prefix hot enough to hit. Conversely a low-volume system whose traffic is bursty — a batch run, a user's working session — hits constantly, because the reuses cluster in time. Always ask about arrival rate *per distinct prefix*, never total QPS. ## How much can you actually lose? Be honest about the magnitude: the downside is **bounded by the write premium**, roughly 25% of the input cost of the cached portion in the common tier, or up to about 100% on a longer-retention tier. You cannot lose an order of magnitude, and only the cached portion of the input is affected — variable input and output are unchanged. That asymmetry against a potential ~90% saving is why caching is worth trying by default on plausibly-repeated prefixes: the upside dwarfs the downside. But "bounded" is not "free", and on a high-volume misconfigured path it is a meaningful and completely silent regression. ## Detecting it Providers report per-request token counts that distinguish cache-creation from cache-read. The health signal is the **ratio**: healthy caching shows read tokens vastly exceeding creation tokens. Creation persistently at or above reads means you are funding writes that nothing collects. Track it per request class, because a single global average will hide one pathological path behind several healthy ones. ## What to do about it The correct response depends on the case. For the genuine one-shot, **turn caching off** for that request class — there is nothing to tune. For traffic below the entry lifetime, either accept the premium as the price of the latency benefit (if a human is waiting and you value fast first tokens more than the surcharge), or stop caching that path. For the unstable-prefix case, the prompt itself needs restructuring so the fixed part really is fixed. ## The senior framing What an interviewer is listening for is that you treat caching as a **per-request-class decision with a measurable downside**, not a global switch that is always on. The judgement is: enable where a large prefix is plausibly reused within the entry lifetime, disable where inputs are unique, and verify with the reported token counts rather than assuming.