A document-QA cache with a five-minute window expires before almost every turn because users return after 40 minutes — longer-TTL tier, keep-warm traffic, or neither?

## The number that decides it The instinct is to compare the window length against the average gap between turns. That is the wrong statistic. What matters is the **cumulative distribution of inter-request gaps for a given prefix**: what fraction of requests arrive within five minutes of the previous one, within an hour, at all. A 40-minute mean is compatible with two very different realities — a uniform 40-minute spacing where no window under an hour ever helps, or a bimodal pattern where 60% of follow-ups arrive within a minute and a long tail drags the mean up. In the second case a short window is already capturing most of the value and the whole problem was a misread average. So the first move in the interview is to say what you would measure: per-prefix gap percentiles, the share of sessions with a second turn, and the size of the stable prefix (a 120k-token document rewrite hurts far more than a 2k system prompt). ## Lever one: a longer window Providers commonly offer an extended window at a higher write premium. It converts a recurring cost — writing the prefix once per gap — into a single more expensive write that covers a longer stretch. It wins when a real share of follow-ups fall between the short and long window, and it loses when sessions are genuinely one-shot, because you have then paid the premium for a write nobody reads. The clean case is a **scheduled batch window**: a job that runs 09:00–10:00 every weekday and nothing else. The entire run fits inside one long window, so one premium write serves the whole hour, and nothing needs to stay warm for the remaining 23 hours. Traffic that is dense inside a bounded window is the single best fit for the long tier. ## Lever two: keep-warm traffic A heartbeat — a small periodic request that touches the prefix just before it would lapse — pins an entry indefinitely by exploiting sliding expiry. Its economics are unusual because the cost does **not** scale with your traffic; it scales with the number of distinct prefixes multiplied by the heartbeat frequency, and it runs whether anyone is using the system or not. That gives a sharp rule. Heartbeats pay when one large, shared prefix serves many users, so a handful of keep-warm requests protects thousands of real ones. They stop paying when prefixes are per-user or per-document, because each one needs its own heartbeat and you are now writing continuously on behalf of users who may never return. They also stop paying overnight and at weekends, when the pinned entry serves nobody — so gate keep-warm on observed traffic, or on the schedule the workload actually runs to, rather than an unconditional cron. The honest check is a count, not a spreadsheet: does the number of keep-warm touches over a period come out below the number of writes those touches avoid? For one shared prefix under intermittent traffic it does, easily. For thousands of per-user prefixes it never does. ## Lever three: do not cache that path Sometimes the workload simply does not have the shape. Single-turn requests each carrying a different document, or genuinely uniform long gaps with per-user prefixes, produce a hit rate near zero however you tune the window. Caching then adds a write premium and operational surface for nothing. Recognising this and saying it is a stronger answer than defending a mechanism that does not fit. ## The fourth move: change the shape The most valuable answer is often none of the three. If per-user document prefixes are what makes caching useless, ask whether the genuinely shared material — the system prompt, the retrieval instructions, the shared policy or schema — can be the part that gets cached, so one warm entry serves everyone while the per-user document sits outside it. That converts a workload with thousands of cold prefixes into one with a single hot prefix, and it is the change with the largest effect on the whole problem. ## Putting it together Map shape to lever: **bursty inside a bounded window** → long tier, no heartbeats. **Steady trickle on a shared prefix** → short window is enough, natural traffic keeps it warm. **Intermittent traffic on one shared prefix** → heartbeat, gated on hours of operation. **Idle-gapped, per-user prefixes** → restructure so the shared part is cached, and accept misses on the rest. Then instrument write-versus-read tokens per route so the decision is revisited when traffic shape changes — because it will, and every one of these choices is a bet on a distribution rather than a permanent property of the system.