When is scale-to-zero right for a self-hosted LLM endpoint?

## Frame it as a trade, not a feature Scale-to-zero is not free efficiency; it exchanges money for latency and for availability. To decide, you need three numbers: the hourly cost of an idle replica, the measured cold-start time, and what the first request after an idle period is worth. A dev-environment endpoint used twenty minutes a day at several dollars an hour is an obvious yes. A customer-facing assistant with a two-second TTFT target is an obvious no. Most real cases sit between, and the analysis is what distinguishes a senior answer from a principal one. ## When it clearly pays - **Internal tools and dev/staging.** Nobody is paying for latency, and idle hours are the overwhelming majority. - **Batch and asynchronous pipelines.** The caller submits work and collects results later, so a four-minute spin-up is amortized over an hour of processing. This is the strongest case: run the fleet hot while the queue is deep, and hold nothing when it is empty. - **Long-tail per-tenant models.** With hundreds of fine-tuned models of which a handful are used daily, keeping each warm is untenable. Note the alternative first: if they share a base model, multi-LoRA multiplexing keeps them all warm on one replica and beats scale-to-zero outright. - **Expensive, scarce-use experiments.** A large model evaluated occasionally is precisely the workload nobody should keep resident. ## When it clearly does not - **Interactive traffic with a TTFT SLO.** A user waiting minutes for the first token has left. No amount of clever queueing hides a cold GPU. - **Traffic that is bursty but frequent.** Arrivals every few minutes cause continuous thrash: the endpoint is always either starting up or shutting down, and you pay the start cost repeatedly while barely saving idle time. Hysteresis — a long idle timeout before scale-in — is the fix, and once the timeout is long enough to avoid thrash, the savings often evaporate. - **Scarce GPU capacity.** If your SKU is intermittently unavailable in your region, releasing a node is a bet that you can get it back. Losing that bet turns a cost optimization into an outage, and the failure is silent until the day it matters. ## The mechanics you must not get wrong Three things trip teams up. **The pod is not the cost.** Deleting a pod frees nothing if its GPU node stays in the pool. Real savings require the node to be released and the instance stopped, which adds node provisioning to the return path — the slowest stage of cold start. Conversely, keeping a warm node pool with no pods running is a middle position: pods start fast because the image and possibly the weight cache are local, but you keep paying for the hardware. **Something must hold the first request.** Stock HPA does not scale below one replica (a scale-to-zero feature gate exists but has long been alpha), so zero comes from KEDA or a serverless/gateway layer that can accept a request with no backend, trigger activation and hold the connection while the replica starts. Whatever holds it must survive minutes, which usually means generous proxy timeouts and a client that will not retry midway — an impatient retry policy against a cold endpoint multiplies the very work you are waiting on. **Cold start must be measured, not assumed.** Everything above depends on that number, and it is the one teams quote from memory and get wrong by a factor of three. ## The intermediate positions that usually win - **A warm floor with burst.** Keep one replica — possibly a smaller model or a smaller GPU — always on to absorb interactive traffic instantly, and scale the expensive tier up on demand. Users never see a cold start; you pay for one replica, not the peak. - **Scheduled scaling.** Most internal traffic follows a known daily and weekly curve. Scaling to zero overnight and back before the workday costs nothing in user-visible latency and captures most of the savings, without betting on a reactive control loop. - **Consolidation.** Before choosing zero, ask whether several sparsely used endpoints could be one endpoint: same base model with adapters, or a shared multi-model server. A shared replica at 40% utilization beats four idle replicas and four cold starts. - **Cheaper idle.** A smaller GPU SKU, a quantized checkpoint or a spot instance can cut idle cost enough that keeping the endpoint warm becomes affordable — often a better answer than eliminating idle entirely. ## The organizational half Whoever chooses scale-to-zero must own the consequence. State it as an explicit contract: this endpoint's first request after idle takes up to N minutes, it is not covered by the interactive latency SLO, and here is the failure mode if GPU capacity is unavailable. That sentence is worth more than the configuration, because it stops the endpoint from being quietly adopted by a latency-sensitive caller six months later — which is how most scale-to-zero incidents actually begin.