Inside a broker node, what do the connection cap, the request queue depth and the handler pool each count?

## Three ceilings in series A node does not have one ceiling on "how busy it is". It has several, on different stages of the same path, each counted in its own unit. Follow a single request: 1. It arrives on a **connection the node is already holding**. If the node is at its connection cap, a new client cannot establish one at all — the request never exists. 2. Once read, the request is accepted and placed in the node's **request queue**, a bounded place for accepted-but-not-yet-processed work. 3. A worker from the **handler pool** takes it, does the real work — appending, reading, answering metadata — and only then is the slot released. Each stage has its own ceiling, and each ceiling is a *count*, not a rate. ## What each one bounds | Ceiling | Unit | What it bounds | Raising it costs | |---|---|---|---| | Connection cap | connections held at one instant | how many clients may be attached at once | memory and descriptors per held connection | | Request queue depth | accepted requests waiting | how large a burst can be absorbed before new work has nowhere to wait | memory, and added waiting time for everything in it | | Handler pool | requests being worked on concurrently | the node's true processing concurrency | cores, memory, and contention between workers | The distinction that matters most is the last two. A **deeper queue is waiting room, not capacity**. It lets a burst land without immediate failure, and it makes each queued request wait longer. It does not cause a single extra request to be processed per second. Only handlers do work. ## Why the separation exists at all - **Reading a connection is cheap; serving a request is not.** Most designs keep the set of workers that read connections separate from the set that does request work, so one slow request cannot stop the node from noticing traffic on other connections. - **A bounded queue is a deliberate choice.** An unbounded one converts a burst into unbounded memory and unbounded waiting; a bounded one makes the ceiling explicit and visible. - **Concurrency needs to be capped independently of arrival rate.** Ten thousand clients may be attached; letting ten thousand requests execute at once on a machine with a few dozen cores is how a node becomes slower under load than it was with less. ## The arithmetic a middle engineer is expected to do Processing concurrency comes from handlers alone, so the node's sustainable request throughput is roughly: ``` requests per second ~= handlers / average service time in seconds ``` With 16 handlers and an average service time of 8 milliseconds, that is about 2,000 requests per second, and no queue depth changes it. The queue only decides how long a burst above that rate can be absorbed before the waiting room itself is full: ``` absorbable burst ~= queue depth x (arrival rate - service rate)^-1 ... in practice: seconds of burst absorbed ~= queue depth / (arrival rate - service rate) ``` That is the whole reason the two are separate settings: one buys time, the other buys work. ## What this looks like from the outside - Requests take longer while the node's processor utilisation is not pinned — work is waiting, and handlers are the scarce thing. - New clients cannot attach while existing ones are served normally — the connection cap is the binding one. - Each of the three is reached independently, and a client that is comfortably inside every bytes-per-second and requests-per-second allowance can still be sitting behind one of them, because none of the three is a rate. What a node *does* at each ceiling — hold the work, answer more slowly, push back on the writer, or refuse — is a separate subject with its own answers, and it differs sharply between platforms. ## Where designs differ Do not assume every platform exposes all three. Some make the handler pool and the reading workers two explicitly sized settings; others size them from the machine and expose neither. Some bound the queue in requests, others in bytes of queued work. A rented service typically exposes none of the three directly and expresses the whole thing as a tier allowance. The **model** is what transfers: a held connection, a bounded waiting place, and a finite set of workers are three different scarcities, and an operator who cannot say which one is binding cannot pick the right lever.