Why does a per-process semaphore fail to bound LLM API concurrency fleet-wide?

## The semaphore that looked like a limit The idiom is everywhere: acquire a semaphore before each model call, release it after, size it to something that felt safe in load testing. It works beautifully on one machine. The failure appears at the deployment boundary, because the semaphore is a property of the *process* and the rate limit is a property of the *account*. A team sets a cap of 40 in-flight calls, tests it against a single instance, sees clean headroom, and ships. Later, someone adds a second worker pod for redundancy, then autoscaling takes the deployment to ten. Nothing in the application changed and no configuration was touched, but the provider now sees roughly 400 concurrent calls. Throttling starts, and the confusing part is that every instance's own metrics look healthy: each is at or below its configured 40. The limit that was violated is not visible from inside any one process. ## The autoscaling feedback loop This failure has a nasty amplifier. Throttling raises per-request latency (calls now wait through backoff). Higher latency means requests spend longer in flight and in local queues. Queue depth and latency are exactly the signals an autoscaler is usually configured to react to, so it adds replicas — and each new replica brings another full semaphore's worth of concurrency to an account that is already over its ceiling. The system scales *into* the wall. Any fleet whose scaling signal is downstream latency needs its provider concurrency governed outside the replica, or it will do this. ## Concurrency is not the quota anyway Even a correctly shared concurrency cap only approximates the thing being metered. Providers meter tokens per minute as well as requests, and per-request token cost varies by orders of magnitude across a real workload: a 200-token classification and a 60,000-token document summary occupy one semaphore permit each. Forty in-flight summaries can exceed a token ceiling that forty in-flight classifications would not approach. Extended-thinking modes widen the gap further, because reasoning tokens count as output and can multiply an answer's token cost several times over with no change in request count. A concurrency cap tuned when a workload ran at low reasoning effort silently becomes far too generous when the effort dial is raised. If tokens are what is metered, tokens are what you should pace against; concurrency is a convenient, coarse proxy that needs a margin. ## Three ways to make the bound real **Static division.** Give each replica `total_budget / replica_count`. It is trivial and needs no shared state, and it is genuinely fine for a fixed-size deployment. Its weaknesses are that capacity is stranded whenever load is uneven (an idle pod's share is unusable by a busy one) and that the divisor is a hidden coupling: a scaling change that nobody thinks of as a rate-limit change quietly invalidates it. **Central metering.** Route model calls through one place — a shared limiter backed by a common store, or an internal gateway every service calls — so the budget is enforced at account shape. This is the version that actually holds under autoscaling, and it brings a second benefit: one place to observe consumption, apply per-workload priority, and change limits without redeploying callers. The costs are a network hop, a dependency that must itself be highly available, and a decision about what happens when the limiter is unreachable (fail open and risk throttling, or fail closed and stop all model traffic). **Adaptive concurrency.** Rather than deriving the cap from a published number, discover it: increase the in-flight allowance gradually while calls succeed, and cut it sharply when a 429 arrives — the classic additive-increase / multiplicative-decrease shape. This handles quota changes, noisy neighbours inside the same account, and heterogeneous request sizes without anyone maintaining a constant. It is more code, it needs a floor so the system does not collapse to one, and it must be per-account rather than per-pod for the same reason the static cap was. These compose. A common production shape is central metering for the account budget, plus a local adaptive cap so an instance backs off promptly on throttling instead of relying on a round trip. ## Sizing, when you must pick a number Little's Law gives the starting point: concurrency ≈ throughput × average latency. To sustain 20 requests per second against calls averaging 4 seconds you need about 80 in flight. Do that arithmetic per workload, then sanity-check it against the token ceiling using the workload's mean tokens per request, and take the lower of the two. Remember that latency here includes provider-side thinking time, which is the term most likely to move. ## What to watch Three signals separate this failure from every other latency problem: throttled responses per minute at the *account* level, concurrency observed in aggregate rather than per pod, and the ratio between the two. A fleet whose per-pod concurrency looks healthy while account-level throttling climbs is showing you exactly this bug, and the fix is to move the bound to where the meter is.