Why does a timestamp at the top of a system prompt destroy cache hit rate?

## The rule that explains the behaviour A prompt cache stores the provider's processed representation of a *prefix* of your prompt and reuses it when a later request starts with exactly the same tokens. Two properties follow, and together they explain the whole failure: 1. **Matching is exact, not semantic.** "Current time: 14:32:07" and "Current time: 14:32:08" are not similar-enough; they are different tokens and the match stops there. Nothing about prompt caching is fuzzy — that is a different mechanism entirely. 2. **Matching is a prefix, so it is order-sensitive and unrecoverable.** The cache can reuse the leading run of identical tokens and nothing past the first difference. Content that is stable but sits *behind* a volatile field cannot be salvaged, no matter how large or how unchanging it is. A per-second clock at the very top of the system prompt is the worst possible placement: it diverges at token ~4, so the run of reusable tokens is essentially zero and every downstream byte — the 20k of policy, the tool definitions, the few-shot examples — is reprocessed on every single call. ## The second, quieter cost It is tempting to think the outcome is simply "caching does nothing". It is worse than neutral. Storing a prefix is not free: the tokens written into a cache entry are processed and stored, and providers price that write differently from an ordinary read. If every request writes a brand-new entry that no subsequent request can ever match, you are paying a surcharge on every call to populate a cache with garbage. This shows up unmistakably in instrumentation as cache-creation tokens tracking cache-read tokens one-for-one, or as creation with essentially no reads at all. ## Recognising the pattern beyond timestamps A clock is the canonical example because it is so visibly per-request, but the same defect appears in many disguises, and all of them share the shape "a value that varies per request has been placed inside the region you wanted to reuse": - A greeting that interpolates the end user's name or account ID into the system text. - A request or trace ID injected for debugging, often added innocently during an incident and never removed. - An A/B flag or feature-toggle snapshot rendered into the preamble. - A random seed, or few-shot examples shuffled per call to "reduce position bias". - A session counter, remaining-quota figure or token-budget line that decrements as the conversation runs. - Retrieved documents pinned above the instructions in relevance order, where the order changes with the query. Each of these looks harmless in a diff and each converts a large reusable block into dead weight. ## The fixes, in order of preference **Hoist it out.** Move the volatile value below everything you want cached — into the variable tail, or into the user turn. The model reads it just as well from there, and the entire stable region above it becomes reusable. This is almost always the right answer and costs nothing. **Let the model ask for it.** If the value is only needed occasionally, expose it through a tool call rather than pushing it into every prompt. The prefix then never carries it at all. **Coarsen it.** When something time-like genuinely has to sit high in the prompt, reduce its resolution to the coarsest value the task tolerates: a date instead of a second, or an hour bucket. Now the prefix changes once per bucket instead of once per request, which costs you a single cold call per bucket and gives you hits for everything in between. Be honest with yourself about what the task needs — a support agent almost never needs second-level precision, and if it does, a tool is the better answer. **Bucket deliberately, not accidentally.** If you coarsen, align the buckets with traffic. An hourly bucket that rolls over at the top of the hour produces one cold request per hour per prefix; that is a rounding error. A bucket that rolls every 30 seconds under bursty traffic is barely better than no bucketing. ## How to find it in an existing system Do not read the prompt looking for the problem — the whole point is that the offending line is easy to miss. Hash the exact bytes of the cacheable region at the call site and log the hash. If a route that should have a handful of prefix shapes emits thousands of distinct hashes per hour, you have confirmed content variation. Then diff two prefixes with different hashes programmatically and the volatile field falls out immediately. If a deploy caused it, the hash cardinality steps up sharply at that release. ## What not to conclude A longer retention window does not help a prefix that changes on every request — nothing is being evicted, nothing was ever reusable. Nor does retrying, nor does asking the provider to "enable caching harder". The problem is entirely in what you sent, and the fix is entirely in your prompt assembly code.