You own a service whose worker pool is periodically overwhelmed. Design its behavior at the submission boundary — how the system should decide what to accept, what to shed, and how that decision reaches the producers — and justify the choices you would make.

## Framing: overload is a designed state Every system has a capacity, and every system will occasionally be offered more than it. The question is never "how do we avoid rejecting?" but "when demand exceeds capacity, what does this system do, and who finds out?" The submission boundary — the moment a task is offered to the pool — is where that contract is expressed. The failure to design it does not remove the decision; it just makes it implicit. An unbounded queue is a decision: *accept everything, degrade latency without limit, die on memory*. That is rarely the decision anyone would make on purpose. ## 1. Bound from a budget Derive the bound: **max acceptable queue wait × measured throughput**. If p99 end-to-end budget is 300 ms, service time is 100 ms, and the pool completes 400 tasks/second, allowable wait is ~200 ms, so the bound is ~80 items. Then check the memory implication (bound × worst-case retained bytes) and take the smaller. Re-derive whenever throughput changes; a bound tuned for last year's service time is arbitrary today. Be suspicious of large bounds. A large bound is not "more headroom" — it is a longer period during which the system is saturated and nobody knows. ## 2. Separate by criticality, not by convenience The single most valuable structural decision is refusing to let unrelated work share a queue. A pool shared between checkout requests and analytics exports has exactly one behavior under load: both degrade together, and the cheap work usually wins because there is more of it. Give each criticality class its own pool and bound (bulkheads). Now overload in one class is *contained*: the analytics pool rejects, checkout is untouched. This costs some efficiency — idle workers in one pool cannot help another — and that inefficiency is what you are buying isolation with. State it that way in an interview; it is the tradeoff, not a flaw. ## 3. Choose the shed policy per class - **Best-effort** (telemetry, prefetch, recompute): drop, and count the drops. Loss is acceptable; blocking the producer is not. - **User-facing, retryable**: fail fast with an explicit overload response, ideally carrying a retry-after hint so the client's backoff is informed rather than guessed. - **Internal pipeline stage with a bounded producer**: block with a timeout. This is real backpressure and it propagates upstream stage by stage; the timeout keeps the stall bounded and reportable. - **Must-run, cannot lose**: it does not belong in an in-memory queue at all. Persist it to durable storage and let a consumer drain at its own rate — the durable store *is* the bounded queue, and its depth is your backlog metric. ## 4. Deadlines and service order Depth bounds *how many* wait; a deadline bounds *how long* any one waits usefully. Stamp each item with the caller's deadline at submission. Workers check it before starting and skip expired items. Under overload this is the difference between a pool doing 100% useful work and a pool doing 100% archaeology. Service order matters too. Strict FIFO under sustained overload makes *every* request slow, since each waits behind the whole backlog. Serving newest-first (LIFO) under deep backlog keeps a subset of requests fast while the old items expire and are skipped — worse fairness, dramatically better usefulness. This is the same intuition as a drop-oldest policy, applied to ordering instead of eviction. ## 5. Close the loop A rejection nobody observes is just an outage with extra steps. Emit, per pool: queue depth, **queue wait percentiles**, rejections/sec, drops/sec, expiries/sec, worker utilization. Wait percentiles are the number that maps to user experience; depth alone is meaningless without throughput. Those metrics feed three consumers: - **Alerting** — sustained nonzero rejection on a critical class is a page. - **Autoscaling** — rejection rate and queue wait are far better scaling signals than CPU, because a pool waiting on I/O can be saturated at low CPU. - **Producers** — the contract must state what a refusal means and require exponential backoff with jitter, a retry budget (retries as a bounded fraction of traffic), and a circuit breaker. Fail-fast without these turns rejection into amplification and produces congestion collapse. ## 6. Push the boundary outward The pool's queue is the *last* place to shed. Cheaper places, in order: rate-limit or quota per client at the edge; admission control that rejects at the front door based on measured latency; a load balancer that stops sending to a saturated node; and degraded modes (cached, partial, or lower-fidelity responses) that cost a fraction of the full path. Shedding early is orders of magnitude cheaper than shedding after a request has consumed connections, memory, and downstream calls. ## What to say when asked for the tradeoff Bounds and isolation cost utilization: you will sometimes reject while some worker somewhere is idle. You accept that in exchange for predictable latency, bounded memory, contained blast radius, and an honest saturation signal. Systems that optimize purely for utilization are the ones that fail all at once.