Which levers actually raise a reader group's drain rate, and where does adding more reader instances stop helping?

Once a drain is understood as needing a surplus over the arrival rate, the practical question is where that surplus comes from. There are only three levers on the reading side, each bounded, and knowing where each one stops is most of the skill. ## Lever one: more members Adding reader instances is the obvious move and the one with the hardest wall. Where a stream is divided into a fixed set of parts and each part is held by exactly one member at a time, the **parallelism ceiling** is the number of parts: a group of thirty members on a stream with twelve parts has eighteen members holding nothing at all. They consume resources, they take part in every redistribution of shares when membership changes, and they contribute zero throughput. How that ceiling is set in the first place is a design decision made long before the incident and is not something an operator can change mid-drain in any cheap way. Where the design has readers competing for records from a shared queue rather than owning parts of a split stream, there is no such ceiling, and members can be added far more freely. The wall then moves somewhere else, and usually to one of two places: a broker-side rate ceiling applied per client or per tenant, which caps the drain regardless of how many readers you start, or the shared downstream system every record ends up writing to. The second is the more common surprise — twenty readers all inserting into one database do not go twenty times faster; they go as fast as that database goes, and then somewhat slower as they contend. ## Lever two: more records per request Taking more records per request amortises the fixed cost of each round trip to the broker across more records, and on a network-bound drain it can be worth more than another handful of instances. It is not free: - **Per-record latency rises**, because a record waits for its batch to fill and for the whole batch to be handled. - **Memory in the reader rises** with the batch, and an over-large batch turns a throughput problem into an out-of-memory problem. - **The unit of lost work grows**: when a member fails mid-batch, the work that has to be redone is a batch, not a record. - **The batch must still finish inside the progress deadline** — the timer that decides a member has stopped making progress. A batch large enough that a member goes quiet for longer than that timer allows gets its share taken away, which costs far more than the batching saved. ## Lever three: waiving in-order handling Where handling is constrained to happen in order, that constraint is often what is suppressing parallelism, and relaxing it for the duration of the drain is a real lever — handling records concurrently within a share, or spreading work more widely than the ordering rule normally permits. It differs from the first two in kind: it changes what the system computes, not merely how fast. It is safe only where the handler's effect does not depend on the order records arrive in, and it is a waiver, which means it is explicit, time-boxed and put back. Treating it as a knob rather than a decision is how a drain turns into a correctness incident. ## The levers side by side | Lever | What it buys | What it costs | Where it stops | |---|---|---|---| | More members | Near-linear throughput while shares remain | Cost; churn while shares settle | The parallelism ceiling, or a shared downstream system | | More records per request | Fewer round trips per record | Latency, memory, a bigger unit of lost work | The progress deadline, and diminishing returns | | Waiving in-order handling | Parallelism the ordering rule suppressed | Correctness, if the handler depends on order | Only where the handler tolerates it | ## The wall that is usually not the broker A drain plan that names only broker-side limits is usually wrong about its own bottleneck. Most reader groups spend the majority of each record's time outside the broker entirely — in a database write, an outbound call, a transformation. Measure the drain rate at completion, find which component saturates first, and add capacity there; otherwise more readers buy nothing but contention. The one lever that never appears on this list is asking the writers to slow down, which is rarely yours to pull and usually pushes the problem one system upstream. ## Deciding before you need it The first two levers are reversible and change no outcome, so they can be pulled by whoever is on call. The third changes results and should have been settled in advance, per stream, by the people who own what the records mean.