Load Testing & Saturation Signals

## Why the distinction matters "Performance testing" is not one activity. The three classic shapes apply different traffic profiles and answer different questions. Picking the wrong shape gets you a green result that says nothing about the risk you were actually worried about — which is worse than no test, because it buys false confidence before a launch. ## Load test: does it meet its targets at expected peak? A load test applies the traffic you expect — the peak of a normal day, or a forecast peak for a known event — and holds it there. Pass criteria come from the service's own targets: p99 latency under the threshold, error rate under the threshold, and no saturation signal trending upward across the hold. Two details decide whether the number means anything. First, **hold long enough to reach steady state**. Runtime warm-up, cache fill, connection pools growing to their working size, and autoscaling all move the numbers for the first several minutes; discard the warm-up and judge the plateau, not the average over the whole run. Second, **the request mix must resemble production**. A test that hammers one cheap cached endpoint measures that endpoint, not the service — real traffic has a mix of expensive writes, cold reads, and pathological large payloads, and the expensive tail is usually what saturates first. ## Stress test: where and how does it break? A stress test goes deliberately past expected peak until something gives. You are not measuring compliance, you are collecting two facts. The first is the breaking point: the offered rate at which latency or errors leave the acceptable band. The second, and more valuable, is the **failure mode**. Does the service shed the excess and stay up with elevated errors, or does it collapse — queues growing without bound, request timeouts cascading into callers that retry, out-of-memory kills, a connection pool wedged so hard it does not recover when load falls? A service that degrades predictably at 3x and recovers on its own is in far better shape than one with a slightly higher ceiling that needs a restart afterwards. That last point is why a stress test should always include the ramp back **down**. Return to normal load at the end and watch whether the service returns to baseline latency unaided. Many systems have a hysteresis problem — they cannot dig themselves out of a queue backlog even once the offered rate is survivable — and that is a much more dangerous property than the ceiling itself. ## Soak test: what only breaks with time? A soak or endurance test runs moderate, realistic load for hours or days. It finds the class of defect that is a function of elapsed time or cumulative request count rather than of rate, and is therefore invisible in a fifteen-minute run: - heap that grows slowly until an out-of-memory kill - sockets or file descriptors never closed, hitting a per-process limit - an in-memory cache with no bound or eviction policy - logs or temp files filling a disk - a metric whose label cardinality grows with unique inputs - credential, token, or connection expiry paths that only execute after hours - memory fragmentation and allocator drift The useful rule: **a soak must outlast the natural cleanup interval.** If the platform restarts instances every six hours, a two-hour soak proves nothing about a leak that takes a day — and it should also prompt the question of whether production is quietly relying on those restarts to stay alive. ## Related shapes worth naming A **spike test** jumps to high load instantly rather than ramping. It finds cold caches, autoscaler reaction lag, connection-pool warm-up cost, and thundering-herd behaviour that a smooth ramp completely hides. A **capacity or ramp test** steps load upward specifically to locate the point where latency turns, which is a different exercise from confirming a target at a fixed rate. ## Choosing, before a specific event The choice has a real cost, so tie it to the risk you are carrying: - A launch with a fixed date and a forecast multiplier wants a load test at forecast peak, plus a stress test so you know how much slack exists if the forecast is wrong. - A service that just added a cache or a new client library wants a soak, because that is where leaks live. - A service fronting a scheduled on-sale wants a spike test, because the risk is arrival *shape*, not total volume. ## Common mistakes Reporting a stress-test breaking point as "capacity" — the breaking point is where it fails, not where it is safe to run. Running against stubbed dependencies that reply in a millisecond, which moves the bottleneck entirely. Not resetting state between runs, so run two inherits run one's warmed caches and looks faster. And treating zero errors at peak as proof of headroom when you never measured how far past peak the cliff sits.