Several pipelines fan out to one shared classification service: how would you decide the concurrency bound each flattening step uses?

## The bound is a capacity decision A flattening step's concurrency bound looks like a local tuning knob and is not. It is the maximum number of simultaneous calls this step places on a shared dependency, and the number that matters is not the one in the code: it is that bound multiplied by the number of running instances of the pipeline, summed over every pipeline that calls the same service. A bound of 16 in a step running on twelve instances is 192 concurrent calls from one caller. Teams discover this during an incident, not during a review. So the decision is made in the opposite direction from the way it is usually approached. Not *how much concurrency does my pipeline want*, but *how much concurrency can the dependency serve, and how is that budget divided*. ## Two numbers to reconcile **The budget (top-down).** Measure the dependency's latency against offered concurrency and find the knee - the point past which latency rises faster than throughput. Set the fleet-wide budget below that knee with headroom for bursts and for whatever is not in the budget yet, then divide it among callers by importance rather than equally. **The need (bottom-up).** Little's Law gives the other number: the items in flight equal arrival rate times the time each spends in the system, so **arrival rate x inner-call latency** is the concurrency required just to keep up. A queue receiving 40 items a second against a 250 ms call needs 10 calls in flight to break even. A share below that means the step is the permanent bottleneck and lag grows linearly forever; a share far above it buys nothing because the arrivals are not there. When the two numbers disagree, that is the real finding - the demand does not fit the dependency - and the answer is a capacity conversation, a cheaper call, or fewer calls, not a larger bound. ## Where the limit lives | placement | what it isolates | utilisation | symptom when it trips | |---|---|---|---| | per flattening step | that one pipeline | lowest - each share sits idle when unused | that pipeline's lag grows, visible in its own metrics | | shared permit pool per process | all pipelines in the process from the outside world | higher - shares are pooled | whichever pipeline asks next waits, so blame is unclear | | at the dependency | nothing on the caller side | highest | failures appear across all callers at once | The useful principle is that you want the limit **you own** to trip first. A bound in your own step fails in a place you can observe, with one symptom you can attribute, and a number you can change. A limit enforced at the dependency arrives as failures spread across every caller and tells you least about which one caused them. Having both is fine, provided yours is the tighter of the two. ## Isolation against utilisation Separate per-pipeline bounds are the isolation choice: a batch backfill cannot consume the share that live traffic depends on, and each pipeline's behaviour is legible on its own. The cost is idle capacity, since one pipeline's unused share is not available to another. A single pooled bound is the utilisation choice and gives that back, at the price of a noisy neighbour being able to take the whole pool. For a shared classifier serving both interactive and batch work, the defensible default is separate bounds with the batch share deliberately small, because the failure you are protecting against is the backfill, and it is the one you can predict. ## What the bound does not do It does not reject work. When the bound is reached the step simply stops subscribing to further inner sources, which in a demand-driven pipeline means it stops requesting elements and the pressure propagates back upstream. Whether excess work should be refused, shed or delayed is a separate decision made elsewhere, and conflating the two leads to a bound set as though it were an admission policy. It also does not protect you from retries. A retry inside the inner call multiplies effective concurrency by the retry factor at exactly the moment the dependency is struggling, so the bound must be sized against the retried volume, not the nominal one. ## Making it a standard rather than a number What a lead owns here is the process, not the integer: 1. Publish the dependency's budget and who holds which share, so the number in any one repository is traceable to a decision. 2. Require every flattening step that calls a shared service to state its bound explicitly, so nothing inherits an implicit default. 3. Make the bound configurable without a deploy, since it must move when latency moves. 4. Alert on in-flight count pinned at the bound, which is the signal that the share is now the bottleneck, and re-derive from Little's Law when it fires. 5. Re-divide the budget whenever a caller is added, because the sum is the thing that must hold, not any individual share.