How do you split one LLM token budget across workloads with different deadlines?

## The budget is a shared resource, and it is small An account's per-minute token allowance is a fixed pool that every workload draws from: the interactive feature, the periodic re-scoring job, the backfill someone kicked off manually. Unlike CPU, it is not fairly scheduled by anything — whoever calls first consumes it. So the default behaviour of an unmanaged system is that the loudest workload starves the most latency-sensitive one, and it does so exactly when demand is highest. Take a claims platform: an adjuster-facing assistant that must answer inside a couple of seconds while someone is on the phone, a re-scoring job that refreshes claim priorities every few minutes, and a nightly classification run. All three share one allowance. When claim volume spikes after a storm, all three spike together, and without an explicit policy the adjuster's request queues behind thousands of re-scores. ## Classify by deadline, not by importance Every workload owner believes their workload is important, so importance is not a usable axis. Deadline is. Three classes are usually enough: - **Interactive** — a human is waiting; the answer is worthless after a few seconds. - **Near-line** — driven by an event; useful within minutes; nobody is blocked. - **Bulk** — driven by a schedule; useful within hours; belongs on the batch endpoint whenever the provider offers one. Moving bulk work onto the asynchronous batch path is the single biggest structural win, because it typically draws on a separate allowance and removes that demand from the contended pool entirely. What remains to be allocated is the interactive and near-line traffic that genuinely needs the synchronous endpoint. ## Reservation versus shedding Two mechanisms, and you need both. **Reservation** protects the floor. Guarantee the interactive class a share of the budget that lower classes cannot consume — a dedicated slice of concurrency, or a limiter that admits bulk work only when interactive consumption is below a threshold. The reserved floor should be sized from the interactive class's own peak, not from its average, because the whole point is to survive the burst. **Shedding** handles the excess. When offered load exceeds the pool even after reservation, something must be refused, and the design choice is *what* and *how visibly*. Refuse the most deadline-tolerant class first, refuse it early (at admission, before it consumes any budget), and refuse it explicitly — an immediate rejection that the caller can retry later is a far better outcome than a request that sits in a queue for two minutes and then fails anyway, having occupied memory and a connection the whole time. The instinct to keep everything and just go slower is the trap. Uniform degradation means the interactive class fails too, so you lose the requests that mattered and keep the ones that did not. ## Queue timeouts: check the deadline on dequeue A queue is a time machine that makes work older. An item admitted with a two-second budget that has waited ninety seconds should be dropped when it reaches the front, not executed. Running it spends real tokens on a result nobody will read, and worse, it delays the item behind it, which may still be inside its deadline. The rule is simple and frequently missed: **stamp each item with its deadline at admission and re-check it at dequeue**. Under sustained overload this converts a queue that would otherwise serve everything late into one that serves recent work on time and discards the rest — which is what the users actually want. For the same reason, prefer serving the *newest* deadline-sensitive work when the queue is deep. A strict FIFO under overload serves the stalest items first, which is precisely backwards. ## Backpressure must reach the caller An unbounded in-memory queue is not capacity, it is deferred failure. It converts an honest rejection into rising latency, growing memory, and a system whose failure arrives all at once and much later. Bound the queue. When it is full, reject at admission with a clear signal, and make sure that signal propagates to whatever is feeding you — an upstream service, a scheduler, a user-facing screen that can say "try again shortly". A caller that receives no signal keeps offering the same load, which is why silent buffering makes overload permanent. ## What is genuinely contested How to allocate between near-line classes has no settled answer. Static reservations are predictable and waste capacity when a class is idle. Weighted fair sharing uses the pool better but makes worst-case latency for any one class harder to reason about. Fully dynamic schemes that reallocate on observed demand are the most efficient and the hardest to debug at 3am. Most teams land on static reservation for the interactive floor — where predictability matters most — and something looser above it, then revisit when the mix changes. What is not contested: an explicit policy, however imperfect, beats the implicit one, which is always "whoever calls first wins, and the person waiting loses". ## Making it operable Three things make the policy real rather than aspirational. Tag every model call with its class at the source, so allocation is possible at all. Report utilisation and shed rate **per class**, so a shed spike is visible and attributable. And rehearse the burst — if nobody has watched the system shed under load, the first observation will be during the incident, and the usual discovery is that some path was never tagged and has been consuming the interactive floor all along.