A two-minute traffic spike is over before any new copy serves a request - where did the time go?

## The chain, stage by stage Reactive autoscaling cannot serve a spike faster than the sum of its own delays. Walk the chain with a concrete set of assumptions - these are illustrative numbers, not defaults, and yours will differ: | Stage | What is happening | Typical cost | |---|---|---| | Measurement window | The signal is an average over a rolling window, so a spike is visible only partly, and late | 30-60 s | | Collection and evaluation interval | The number is scraped or pushed on a period, and the loop runs on its own period | 15-60 s | | Decision to desired count | The recompute plus any gate that requires the reading to hold | 0-30 s | | Placement and start | A host with capacity is found, the image is pulled if not already on that host, the process starts | 10-120 s | | Readiness and warm-up | The copy reports ready and is added to routing; caches fill, connections are established | 5-120 s | Even a well-tuned chain lands around two minutes, and an untuned one around five. Against a spike that lasts two minutes, the capacity arrives to serve traffic that has already gone. ## The averaging window is the sneakiest term An average over a rolling window does two things to a spike, and engineers usually only account for the first: 1. **It delays it.** The window has to fill before the reading reflects the new level. 2. **It shrinks it.** A 60-second window that is half through a spike reports roughly halfway between the old level and the new one. The loop therefore computes a *smaller* ratio than reality justifies, asks for fewer copies than needed, and only reaches the right count an evaluation or two later - by which time the spike may be over. That damping is deliberate: the same averaging is what keeps ordinary noise from jerking the count around. It is the first half of a genuine trade-off, and the second half shows up on the way down as oscillation. ## Readiness is where the last, invisible minute hides A started copy is not a serving copy. It is routed to only when its readiness check passes, and that check is the right place for the warm-up to be reflected: an empty in-process cache, a connection pool that opens lazily, a large configuration or model loaded at start. A copy that reports ready at second three and then serves its first hundred requests slowly has simply moved the warm-up cost onto users instead of into the chain. Both cost the same wall-clock time; only one is visible on a dashboard. ## What actually shortens it Measure each stage before tuning any of them - the fix is worth nothing if it is not applied to the dominant term. - **Shorten the measurement window and evaluate more often.** Cheapest lever, buys tens of seconds. The cost is noise: a twitchier signal makes the count oscillate, which is what a settling window then has to damp. - **Make start-up cheap.** A smaller image pulls faster; an image already cached on the host does not pull at all; a process that opens its connections in parallel rather than serially reaches ready sooner. - **Reduce the warm-up, or admit it.** Pre-load what the copy needs before reporting ready, so the platform's own view of "ready" is honest even though it takes longer. - **Absorb the first minute with capacity that is already running.** This is the only real answer for a spike shorter than the chain. Either the fleet already has spare room at its steady-state target, or a floor of warm copies exists for this purpose. How much of that to hold is a capacity-planning decision, owned elsewhere - but recognising that *no reactive loop can beat its own lag* is the reasoning this question is testing. - **Turn the spike into a backlog.** Where the work is queued rather than served synchronously, the spike becomes a temporarily longer queue instead of a wall of failures, and late capacity still does useful work. That is the structural reason queue-fed workloads tolerate reactive scaling far better than request-serving ones do. ## The judgment underneath The interviewer is usually checking one thing: do you know that autoscaling is **reactive**, and therefore that its value is in tracking load over minutes and hours - the daily curve, the weekly one, a sustained shift - and not in catching a two-minute burst? A team that responds to this incident by making the loop more aggressive usually trades a spike problem for a flapping problem and still misses the next spike. The stage-by-stage measurement is what turns that instinct into a decision.