A service uses a counting semaphore to cap how many requests are in flight to a downstream dependency at any moment. What failure modes would you design against in that limiter?

## What the primitive actually guarantees A counting semaphore guarantees exactly one thing: the number of threads that have acquired but not released never exceeds the initial permit count. Every failure mode below is a way the surrounding code breaks the assumptions that make that guarantee useful. ## 1. Permit leaks The acquire/release pair must bracket the guarded work on **every** exit path — normal return, exception, early return, cancellation, timeout, panic. If a permit is not returned, it is gone for the life of the process. Leaks are cumulative and monotonic: the limiter of 20 becomes 19, then 12, then 0, and the service stops serving with no error anywhere near the cause. The symptom is a slow decay to a hang over hours or days, which is why it survives testing. Mitigation is structural, not disciplinary: acquire immediately before a `try`/scope-guard whose cleanup releases, or wrap the whole thing in a helper that takes a callable, so no caller can write an unbalanced pair. A cancellation-aware runtime adds a subtlety — if a task is cancelled *while blocked in acquire*, the implementation must guarantee it did not consume a permit; if cancellation can race with a hand-off, an interrupted acquire that actually took a permit becomes a leak. ## 2. Double release / release without acquire The mirror image. Semaphores have no ownership, so an extra release simply increments the count and permanently raises the ceiling above your configured value. A retry loop that calls cleanup twice, or a `finally` release plus an error-handler release, silently converts a limit of 20 into 21, then 25. Nothing throws, and the effect only shows as unexplained downstream overload. Guard with a per-acquisition released-once flag, or by making the release path unreachable more than once by construction. ## 3. Unbounded blocking A plain `acquire()` waits forever. When the dependency stalls, permits stop returning, every caller parks, and the outage propagates upward as a hang — the worst failure shape, because hangs consume caller resources and defeat their timeouts too. Always prefer a bounded acquire: try for a small budget, then reject with an explicit "downstream at capacity" signal. The rejection is a working backpressure signal; the hang is not. The acquire timeout should be small relative to the caller's own deadline, because time spent queueing is time not spent on the call. ## 4. The queue in front of the limiter The cap bounds concurrency, not arrival rate. If arrivals exceed the rate the limiter can retire work, the *wait queue* grows without bound: memory climbs, and by the time a request reaches the front its caller has already given up, so the work you eventually do is worthless — the classic queueing-latency collapse. Two mitigations: bound the number of waiters as well (reject immediately once the queue is full — this is what a bulkhead does), and prefer LIFO or deadline-aware admission under overload so at least the freshest requests are still useful. Little's law makes this concrete: if the mean service time is W and you allow N in flight, the sustainable throughput is N/W; anything arriving faster queues forever. ## 5. Starvation and fairness Unfair (barging) semaphores let a thread calling acquire at the instant a permit is released take it ahead of threads already queued. Throughput is better; the tail is worse, and under sustained saturation an individual waiter can be passed over indefinitely. If you have a p99 obligation, choose a fair semaphore and accept the hand-off cost, or bound waiting with the timeout in point 3 so "starved" becomes "rejected", which is at least observable. ## 6. Scope: the cap is local A process-local semaphore caps that process. Deploy R replicas and the dependency sees up to R × N concurrent calls; autoscale the fleet and the dependency's load scales with it, which is exactly backwards — the point of the limiter was to protect something that cannot scale on demand. Either divide a global budget across replicas (and re-divide when the replica count changes), or move the limit to a shared coordination point, accepting that a distributed limiter buys accuracy with latency and a new dependency. Also beware nesting: a global limiter plus per-endpoint limiters can deadlock if a task holds one permit while acquiring another and the acquisition order is inconsistent. ## 7. Not observable = not operable Export permits available, permits in use, current waiters, wait time distribution, acquisition timeouts and rejections. Without "permits in use" you cannot distinguish a genuinely busy limiter from one that has leaked its permits away; without waiter count you cannot see the queue that is about to eat your memory. A leak detector is simple: available + in-use should equal the configured cap at all times. ## 8. What a semaphore is not It limits **concurrency** (things at once), not **rate** (things per second). If the downstream constraint is requests per second, a semaphore is the wrong primitive — you want a token bucket or leaky bucket. The two are related by Little's law (rate ≈ concurrency / latency), which means a concurrency limiter's effective rate silently drops when the dependency slows down. That is often the behaviour you want — it self-throttles under stress — but it is a property to state deliberately, not to discover.