How do you serve one LLM for interactive chat and an overnight 900K-listing scoring job?

## Two workloads, one model, opposite objectives A consumer chat surface and an overnight job that scores 900,000 product listings can run the identical model and still have almost nothing in common operationally. The chat path is judged on **tail first-token latency** and **sustained per-token rate**. Both are user-perception metrics, both must hold during arrival spikes, and both are violated by queueing. Meeting them requires deliberately leaving capacity idle, because a pool sized to average demand will queue during peaks and blow the tail. The scoring job is judged on **tokens per accelerator-hour** and, equivalently, **cost per listing**. No human waits on any individual response, so first-token latency is meaningless and even a multi-minute queue wait per item costs nothing. What matters is finishing within the window at the lowest cost, which argues for the highest concurrency the memory will support, sitting far past the knee that would be unacceptable for chat. ## Why one shared configuration is the wrong answer Run them together and each pays the other's bill. A large backlog of bulk items in the queue puts every interactive request behind a wall of work, so p99 first-token latency for chat degrades exactly when the batch job is most productive. Conversely, holding the shared pool to the chat objective caps concurrency far below what the bulk job could exploit, so the batch runs slower and costs more than necessary. There is no single setting that is right for both; that is precisely what "a frontier" means. The standard structure is therefore two deployments of the same weights, each tuned to its own end, with independent objectives and independent dashboards. The chat deployment: modest concurrency, burst headroom, alerting on p95 and p99 first-token latency and on token rate. The bulk deployment: concurrency pushed to the memory limit, alerting on throughput and completion-by-deadline, with no latency objective at all. ## Scheduling as a cost lever Interactive demand is diurnal; the fleet you size for peak chat is largely idle at 03:00. Running the scoring job in that trough is the single cheapest optimisation available, because it converts already-paid-for capacity into work rather than buying new capacity. This is why the job is described as overnight in the first place. Where capacity is elastic rather than owned, the same logic becomes a scheduling decision against price rather than against idle hardware. ## Sizing and budgeting the bulk job Estimate before you run. Total tokens is roughly 900,000 x (input tokens per listing + expected output tokens). Divide by the measured tokens-per-accelerator-hour of the bulk configuration to get accelerator-hours, then add margin for failures and re-runs — at this scale a small percentage of retries is thousands of items. Trim the prompt: a 200-token reduction in a shared prefix is 180 million tokens across the run, which is a real line item. And cap output length aggressively, because output tokens are both the slowest to produce and the ones that multiply straight into total time. Checkpoint progress so an interrupted run resumes rather than restarts, and sample-evaluate the first few thousand outputs before committing the remaining 895,000 — the expensive failure at this scale is discovering after eight hours that the prompt was subtly wrong. ## When sharing is defensible Isolation is not free. Two pools means two sets of idle headroom and two sets of operational surface. At small volume — a bulk job that is a few percent of daily tokens, or an organisation with a handful of accelerators — a single pool with strict admission control can be the right call: cap bulk concurrency to a fixed slice, give interactive traffic strict priority in admission, and accept that the bulk job runs slower. What makes this work is enforcement, not intent; "we'll run the batch when it's quiet" without an enforced cap reliably becomes an incident. This is genuinely contested ground and depends on scale, on whether capacity is owned or rented, and on how much the interactive tail is worth. What is not contested is that the two workloads have different objectives, and that a design which does not name them separately is a design that will fail one of them silently. ## What an interviewer is listening for That you separate the objectives before proposing any mechanism; that you name cost per unit of work as the bulk metric rather than reaching for latency; that you understand idle interactive capacity as a resource the batch can consume; that you size and sample before committing 900,000 items; and that you can argue honestly for sharing at small scale instead of reciting isolation as dogma.