When does an LLM provider's batch API beat synchronous calls for bulk work?

## Two endpoints, one model Most providers offer the same models through two front doors. The synchronous endpoint answers one request now. The batch endpoint accepts a large collection of requests, queues them, and runs them when the provider has capacity to spare — usually within a window of up to 24 hours, at roughly a 50% discount per token, and typically metered against a separate allowance so bulk work does not consume the same per-minute ceilings your interactive traffic depends on. This is the cleanest cost lever in the toolkit and the most under-used, because teams reach for the synchronous endpoint by habit and then spend their effort fighting rate limits they did not need to touch. ## What you are actually trading Throughput for per-item latency. On the synchronous path, an item's latency is roughly the model's response time, and total throughput is capped by how many calls you can keep in flight inside your rate limits. On the batch path, an item's latency is effectively the whole job's completion time — results are collected when the batch finishes, not as each item lands — while total throughput is set by the provider's scheduler rather than your concurrency and quota. The economics follow the same shape: half price, plus a large saving in engineering effort, because you are no longer building and tuning a fan-out that has to stay inside a token ceiling, back off on throttling, and survive partial failure. ## When batch is the right call The test is whether any *item* has a deadline, or only the *job*. Consider an insurer that must classify 380,000 first-notice-of-loss claim narratives every night so the adjuster queues are correctly prioritised when they open at 3am. No individual narrative needs an answer within seconds; the collection needs to be done by a fixed hour. That is a textbook batch workload: submit in the evening, collect before the cutoff, pay half, and leave the synchronous quota entirely free for adjusters working live claims. The same shape covers backfills over historical records, periodic re-scoring, bulk metadata enrichment, offline evaluation runs, and anything driven by a scheduler rather than a user. ## When it is the wrong call Anything interactive. Also anything whose *next* step depends on the answer within the same window — a multi-step agent loop cannot batch its steps, because step two's prompt is not known until step one returns. And anything whose input goes stale inside a day: pricing against a live market, fraud decisions on an in-flight transaction, triage of an active incident. If the answer would be wrong or worthless by the time the window closes, the discount buys nothing. ## Operating a batch job: partial completion is normal The critical operational fact is that the window is an upper bound on how long the provider will keep trying, not a guarantee that everything finishes. A submission of 380,000 items can come back reporting, say, 94% complete, with 22,000 items expired or failed. Individual items can also fail on their own merits — a malformed request, an oversized prompt, a content refusal — independently of the batch's overall state. So the pipeline needs three things the synchronous path never made you build: 1. **A reconciliation step.** Match returned results back to submitted item IDs and compute the set difference. Never assume the result file is complete because the job reported "done". 2. **A re-drive path.** The remaining items go through the synchronous endpoint at bounded concurrency, or into a second batch, depending on how much time is left before the real cutoff. Size this path for the tail you actually observe, not for zero. 3. **Lead time.** If the business cutoff is 3am and the window is 24 hours, submitting at 9pm is gambling. Submit early enough that a slow batch plus a full re-drive still lands before the cutoff, and alarm on completion percentage rather than on job status alone. ## Sizing the re-drive The re-drive is where batch workloads meet rate limits. Twenty-two thousand items pushed through the synchronous endpoint at unbounded fan-out is exactly the shape that trips throttling and, worse, competes with the interactive traffic you were protecting. Give the re-drive its own bounded concurrency, run it at a lower priority than user-facing work, and let it take as long as the remaining time allows rather than as fast as the client can go. ## The judgement to show Interviewers are looking for the explicit trade rather than the feature knowledge. The strong answer names the axis — per-item latency sacrificed for throughput and cost — identifies which workloads sit on which side of it, and then, unprompted, raises partial completion and lead time, because that is the part that turns a working prototype into a pipeline that meets a cutoff every night.