How do you tell whether the load generator, not the system under test, is the bottleneck in a run?

### The rig is part of the experiment A performance result is a claim about the system under test, and that claim is only valid if the generator was capable of producing the workload it was asked for. Generators are ordinary software running on ordinary machines and they saturate like anything else. When they do, the run reports a plateau — and the plateau is the rig's, not the service's. Worse, the failure is quiet: most generators do not shout that they fell behind. ### The signals, roughly in order of usefulness **Offered versus achieved rate.** The single most valuable chart. If the plan called for 460 requests per second and the run delivered 383, nothing downstream can be interpreted until that gap is explained. In a population-driven run the equivalent check is achieved iteration rate against what the population and think time predict. **Generator host resources.** Processor saturation across the generator's cores is the common one, especially when each simulated user costs a thread, or when response bodies are parsed, correlated or checksummed. Memory pressure and allocation churn slow the issuing loop. Watch the generator's own scheduling delay if it exposes one — the time between when a request was due and when it was actually issued is the most direct evidence there is. **Connection and port limits.** A host has a finite range of ephemeral ports and a finite descriptor limit, and short-lived connections hold ports in a waiting state after close. A run that climbs smoothly and then flattens at a suspiciously round concurrency, with connection errors appearing, is usually hitting one of those ceilings rather than the service's. **Network path.** The interface, the virtual network, or an intermediary between rig and service can saturate long before the service does — particularly with large payloads. Compare bytes per second against the interface's rated capacity, and remember that a proxy or gateway on the path is also a shared resource. **Timing gap.** Compare latency measured at the generator with latency measured inside the service. Some difference is real network and framing cost. A difference that grows with load is a queue somewhere between the two, and if the service's own timings stay flat while generator-side latency climbs, the queue is on the rig side. ### The two experiments that settle it **Scale the rig, not the load.** Split the same workload across two generator hosts. If throughput and latency stay the same, the rig was not the limit. If the result improves, it was. **Scale the load with the rig.** Add a second host and raise the offered load proportionally. If total achieved throughput rises roughly linearly, you have not yet found the service's ceiling; you were looking at the rig's. A third, cheaper sanity check: point the generator at a trivial endpoint on the same service that does almost no work. The rate it achieves against that endpoint is an upper bound on what your rig can produce at all. ### A worked example An insurance quote engine is exercised nightly for 6 hours. Over successive releases the report shows the same ceiling, 291 quotes per second, with a p99 that looks stable. The service's own dashboards show processor use around 38% and a request queue that never grows — a combination that should already prompt suspicion, because a service at its limit usually shows *something* saturating. Instrumenting the rig shows the generator host pinned at 97% processor use, spending most of it parsing and validating the quote response payload for every request. Splitting the identical workload across two rig hosts moves the ceiling to 574 quotes per second and the service finally starts to show its own pressure. The post-mortem also surfaced an off-by-one boundary in the rig's connection-pool sizing: the pool was created with one fewer connection than the configured concurrency, so one simulated user per host was always waiting on a connection. Small on its own, but it is a reminder that rig defects distort the shape of the result, not merely its magnitude. ### Why it matters beyond the wrong number A saturated generator does not just under-report throughput; it changes the character of the measurement. Requests wait in the rig before they are sent, which is precisely the condition under which latency measured from actual send time understates real waiting, and an arrival-rate-driven workload silently degenerates into a self-throttling one. So a rig capacity check is not housekeeping — it is a precondition for the validity of every latency statistic in the report. Publish rig utilisation alongside the results, and state the headroom the rig had, so a reader can see the result was not the rig speaking.