You must choose the permit count for a semaphore that bounds in-flight work against a shared dependency. Walk through the reasoning you would use to derive a number rather than guess one, and how you would revisit it as conditions change.

## Framing: what the number means A permit count is a statement about *concurrency* — how many units of work may occupy the dependency at once. It is not a rate, not a queue length and not a thread count. Getting the number right is a small piece of applied queueing theory plus a capacity negotiation with whatever sits downstream. ## Step 1 — Little's law gives the floor Little's law holds for any stable system regardless of arrival distribution or service discipline: `L = λ × W` - **L** — mean number of units in the system (here: permits held). - **λ** — mean arrival/completion rate. - **W** — mean time a unit spends in the system (the whole time a permit is held, including network, queueing inside the dependency and deserialisation — not just the dependency's internal service time). Read it two ways. Forwards: to sustain 500 requests/s where each holds a permit for 40 ms, you need at least `500 × 0.04 = 20` permits; with fewer, arrivals exceed departures and the queue grows without bound. Backwards: given N permits and a measured W, the ceiling on throughput is `N / W`, regardless of how many callers you add. That backwards reading is the honest way to tell a product team what a limit costs them. The most common mistake is measuring W as the dependency's reported service time rather than the full permit-hold time. Every millisecond between acquire and release counts, including retries and connection setup. ## Step 2 — Queueing theory says don't size for full utilisation For a simple M/M/1-ish model, mean waiting time scales as `W ∝ 1/(1 − ρ)` where ρ is utilisation. At ρ = 0.5 latency is roughly double the service time; at 0.9 it is ten times; at 0.99 it is a hundred. Real systems are worse than the model because of variance and coordinated arrivals. Consequences: - Size for a target utilisation of roughly 0.6–0.8 of the dependency's measured saturation point, not 1.0. The last 20% of capacity costs unbounded latency. - With **c** parallel servers behind the dependency (M/M/c), pooling helps: c servers at utilisation ρ queue far less than c independent single-server queues, which is an argument for one shared limiter over many small per-caller ones. ## Step 3 — Find the dependency's real ceiling empirically The theory tells you the shape; only a load test tells you the constant. Ramp concurrency and plot throughput and p99 latency against it. Throughput climbs, then flattens, then latency knees upward while throughput is flat or falling — that knee is the dependency's usable concurrency. Beyond it you are adding queueing, not work; past it many systems collapse (thrashing, GC pressure, connection exhaustion, retry storms). Set the global budget at or below the knee. ## Step 4 — Split the budget across replicas and callers A process-local semaphore caps one process. What the dependency experiences is `permits_per_process × replicas × callers`. So the derivation runs downwards: total safe concurrency C at the dependency → your service's share (negotiated, since other clients exist) → divided by your replica count → permits per process. This makes the number fragile in exactly one direction: autoscaling. Double the replicas to handle load and you double the pressure on the dependency that was already the bottleneck. Options: re-derive permits as `share / replicas` dynamically from the actual replica count; keep per-replica permits low enough that the maximum fleet size is still safe; or move the limit to a shared coordination point and accept the extra latency and failure mode. Adaptive limiters (which infer the concurrency limit from observed latency gradients, the way congestion control infers bandwidth) are the third option and are worth naming. ## Step 5 — Decide what happens at the boundary The permit count only defines the cap; the policy defines the behaviour at the cap. Choose deliberately: - **Shed** — bounded acquire, then reject. Preserves latency for admitted work; the caller sees a clear signal. - **Queue** — allow waiting, bounded in both count and time. Smooths bursts, but every queued millisecond is latency the client pays. - **Degrade** — serve a cached or partial result rather than reject. Under overload, LIFO or deadline-aware admission beats FIFO, because FIFO preferentially serves requests whose callers have already timed out. ## Step 6 — Variance, not just means Little's law is about means. If W has a heavy tail — an occasional 2-second call among 40 ms ones — then a permit count sized on the mean will be fully occupied by tail calls exactly when load is highest. Sanity-check the number against p95 W, and cap the hold time itself with a call timeout so a single stuck call cannot occupy a permit indefinitely. A permit held forever is a permanent reduction in your limit. ## Step 7 — Instrument and re-derive The number is a hypothesis with a shelf life. Export permits in use, waiters, wait time, rejections and the measured W; alert when utilisation sits above target or when rejections appear without a corresponding downstream problem. Re-derive after any change to W (a downstream deployment, an added network hop, a schema change) or to replica count. Treating the constant as permanent is the actual failure — the derivation, the measurements and the review cadence are the deliverable, not the integer. ## What a strong answer sounds like A number, the law it came from, the measurement it depends on, the utilisation headroom, the fleet-wide multiplication, the overload policy, and the trigger that would make you change it.