How do you keep a 200-worker agent fleet reusing one cached 30k-token prefix?

## Frame the problem correctly first "200 workers share a 30k-token policy prefix" is not one problem. It is two, and conflating them is the most common way to answer this badly. **Who holds the cached prefix?** With a hosted provider, the entry is generally scoped to your account and model configuration, so any worker's request can match a prefix another worker created — the fleet is already sharing. With self-hosted or replica-based serving, the processed prefix lives in the memory of one serving replica. Nothing is shared implicitly; a request that lands on a different replica finds nothing. **When does the entry become readable?** In both worlds, an entry has to be written before it can be read. A cold fan-out of 200 simultaneous requests, all carrying a prefix that has never been sent, has no entry to match — they race, several pay the write, and the reads only start arriving for whatever fraction is scheduled after a write completes. ## The cold stampede and the warm-up call The practical consequence: the shape of your traffic ramp determines your hit rate on the first minute of a burst. If a scheduled job wakes 200 workers at the top of the hour on a freshly deployed prompt, that first wave is largely uncached, and you will see a spike of cache-creation tokens followed by a normal read pattern. A warm-up call fixes it cheaply. Before releasing the burst, send **one** request carrying the exact reusable prefix and wait for it to complete. It can be a trivial request — the point is the write, not the answer. Then fan out. The rules that make this work in practice: - The warm-up prefix must be **byte-identical** to what production sends. A warm-up built by different code, or with a slightly different tool order, warms an entry nobody will match — a classic and embarrassing failure. - It must **complete** before the fan-out; firing it concurrently defeats the purpose. - It must be **repeated** whenever the prefix changes: after each deploy that touches the prompt, and periodically if a route's traffic is sparse enough that entries go idle between bursts. - Ramping — a small first wave, then full concurrency — achieves the same effect without a dedicated call, and is a good default for autoscaling workers. ## Routing and affinity when the cache is replica-local On self-hosted serving, prefix reuse is a *scheduling* problem. Standard round-robin or least-connections balancing actively destroys it: the same logical prefix gets warmed independently on every replica, multiplying prefill work by the replica count and wasting the memory that holds those entries. Prefix-aware routing (often called cache-aware routing) is the answer: compute a hash of the reusable prefix at the client or router, and prefer the replica that has recently served that hash. This is the same shape as consistent hashing in any sharded cache, with the same costs: - **Hot shards.** A popular prefix concentrates load on one replica. Allow a prefix to spread across a small set of replicas rather than exactly one, sized by that prefix's traffic. - **Balance versus reuse.** Strict affinity can queue requests behind a busy replica while others idle. Route on prefix *until* the target crosses a utilisation threshold, then fall back to least-loaded and accept a cold call. Reuse is an optimisation; latency SLOs are not. - **Churn.** Scale-up, scale-down, restarts and drains all move entries. The router must tolerate a miss gracefully rather than pinning to a replica that no longer holds anything. ## Batching and grouping related work Even where the cache is account-scoped, grouping still pays. Requests sharing a prefix should be run close together in time rather than interleaved with unrelated work, because entries are kept alive by use — sparse, scattered traffic on a prefix lets it go cold between calls, and each revival costs a write. Concretely: sort or partition a queue by prefix identity so a worker pool drains one prefix family before moving to the next, and prefer a dedicated pool per prompt family over a general pool that mixes them. This is also why a fleet with three prompt variants running A/B often shows worse aggregate hit rate than either variant alone would — the traffic is split three ways. ## What to measure and when to stop Instrument cache-read versus cache-creation tokens per route and per replica, and watch creation spikes at burst boundaries — that is the signature of an unwarmed ramp. On self-hosted serving, also watch the distribution of a given prefix hash across replicas; if one hash appears on every replica, affinity is not working. And be willing to conclude it is not worth it. Affinity routing adds a stateful component to what was a stateless tier, complicates draining and autoscaling, and can degrade tail latency. For a small prefix, or a fleet whose traffic is dense enough that entries stay warm naturally, plain warm-up and prefix stability capture most of the benefit with none of the operational cost. Reach for affinity when the prefix is large, the fleet is wide, and the measurement shows replicas warming the same prefix repeatedly.