How do you tell queueing delay from service time when a performance run recorded only end-to-end duration?

## The two kinds of time inside one number A request's end-to-end duration contains exactly two kinds of time: time spent being worked on, and time spent waiting for something that was busy with someone else. - **Service time** is the cost of doing the work once the request holds whatever it needs, with nothing competing for it. - **Queueing delay** is the interval between the request being ready to be served and actually being served. Every layer on the path has its own pair. There is a slot to acquire before a worker runs the request, a connection to acquire before an outbound call leaves, a place in line at a shared store. What was recorded is the sum of all of them, and a sum can never be split by staring at it. You separate the components by **changing one thing and watching which part of the sum moves**. ## Get a service-time baseline first The cheapest useful observation is the operation on its own: one request at a time, repeated twenty or so times, in the same environment and against the same data as the loaded run. With nothing else in flight there is nothing to wait for, so what you record is service time plus the fixed cost of getting bytes there and back. Record the middle of that sample and its spread — repeated single requests should land close together. That number is your yardstick. Under load, the amount by which a request's duration exceeds it is waiting, subject to one caveat below. ## Read the response to concurrency Now hold the run at a ladder of increasing concurrency levels — several minutes at each, measuring only the settled portion — and record duration and completed work per second at every level. The two components respond to concurrency in opposite ways, and that is what makes them separable. | Observation as concurrency rises | Queueing-dominated | Service-dominated | | --- | --- | --- | | Median duration | rises roughly with requests in flight | stays flat | | Completed work per second | flattens at a ceiling | rises with concurrency | | Spread of durations | slow end stretches further each level | stays tight | | Arrival rate multiplied by duration | tracks the number in flight | stays low | The last row is **Little's Law** used as an arithmetic check rather than a prediction: the average number of requests inside a system equals the arrival rate multiplied by the average time each one spends inside it. Two of those three quantities are almost always recorded, so the third is not a matter of opinion. When measured duration climbs while completed work per second refuses to, essentially all of the added time is queueing — the system is finishing the same amount of work and simply making each request wait longer for its turn. ## The caveat: service time itself can move Subtracting the single-request baseline is only honest if the work per request did not change. Two things break that assumption: 1. **Contention inside the work.** If requests fight over something shared — an exclusive section over one record, one refillable cache entry, one counter — the work itself genuinely takes longer when requests overlap. That is not queueing ahead of the resource; it is service time that grew. 2. **A different mix.** If the loaded run exercises a heavier blend of operations than the baseline did, the two numbers describe different work and the subtraction is meaningless. The first is worth naming explicitly rather than folding into wait, because the fix is different. Wait ahead of a resource is relieved by more servers, more slots, or fewer arrivals; work that got slower under overlap is relieved only by shrinking or removing the shared section. ## Why the distinction decides what to do next This is not a taxonomy exercise. The two diagnoses point at opposite remedies, and applying the wrong one is a common and expensive mistake: - If the time is **service**, the work must get cheaper or the resource faster. Adding parallelism multiplies the same cost across more requests and does not reduce any single request's duration. - If the time is **wait**, the resource behind the queue is often sitting with spare capacity while durations climb. Making that resource faster helps a little; increasing how many requests may be served at once, or reducing how many arrive, helps a lot. One more practical point: apply the same reasoning **per layer**, not just end to end. A request whose end-to-end time is dominated by wait may be waiting at one specific layer while every other layer is idle. Recording, for each layer, both the time to acquire the resource and the time to do the work once acquired turns the whole exercise from inference into direct reading — and it is far cheaper to add that pair of measurements before the next run than to argue about a single summed number afterwards.