A node reports wait time and service time per request — what does each measure and what does a rise in each mean?

## The two halves of one request A broker node does not handle a request the instant it arrives. The common shape is: the connection layer reads the request off the socket, puts it in a **request queue**, and a bounded pool of **handlers** takes requests from that queue and does the work — appending records, reading them back, answering a metadata question. That gives one request two clocks: - **Wait time** — from being enqueued to being picked up by a handler. Nothing is happening to the request; it is waiting for capacity. - **Service time** — from pickup to the response being ready. This is the work itself. Their sum is roughly what the client waited inside the node, excluding the network in both directions. ## What a rise in wait time is telling you Wait time rises when requests arrive faster than handlers free up. The causes sit on either side of that sentence: - **More arrivals** — a new client, a burst, a reader group that just restarted and is re-reading, a retry storm amplifying the original rate. - **Fewer free handlers** — the pool is occupied. And this is the coupling to watch: handlers held for a long time by *slow work* push wait time up for every unrelated request behind them, so a service-time problem in one stream shows up as a wait-time problem for everyone. The practical reading is that wait time is a **shared-resource** number. When it moves, the whole node's clients are affected, including ones doing nothing unusual. ## What a rise in service time is telling you Service time rises when the work per request got more expensive, which is much more specific: - reads that used to be answered from memory now come off the volume, because readers fell far enough behind that the records they want are no longer resident; - requests carry more data than they did; - on platforms where an accepted write is held until other copies hold it, that wait is inside the write's service time, so a lagging copy appears here rather than in the node's own work; - the node is contending for its own resources with something else on the machine. ## Where held requests land, and why you must check A request that is deliberately parked — a read told to wait until records arrive or a size threshold is met, or a write waiting on other copies — is not being handled and is not queueing for capacity either. Platforms count that time differently: inside service time on some, in a separate held bucket on others, and on rented services often nowhere you can see. A long-poll read that waited because *no records existed* is healthy; counted naively it looks like a very slow request. Establishing where held time lands is the first step before reading either half. ## Reading the pair together | Wait time | Service time | Most likely reading | |---|---|---| | High | Normal | Not enough handling capacity for the current arrival rate | | Normal | High | The work itself got more expensive for this request kind | | High | High | Slow work is occupying handlers and backing up everything behind it | | Normal | Normal, but clients report slowness | The delay is outside this node — network, the client's own queue, or further down the path | ## What varies across platforms - Not every implementation exposes both halves. Some report only a total duration; some report the queue depth but not the waiting time; rented services often publish neither. - The pool that matters may not be a single one. Some designs separate the threads that read from sockets from those that do the work, and either can be the constrained one. - Where a platform can add handling capacity dynamically, a wait-time rise may resolve itself; where the pool size is fixed at start, it will not. ## What the split cannot tell you It localises the delay to a half of one hop, and no further. It does not say which client or stream caused it — that needs the same numbers split by dimension. It does not say what the node will do if the pressure continues; that behaviour is a separate subject. And it says nothing about the time before arrival or after the response, so a healthy split is not evidence that the whole path is healthy.