To make sure no request is ever rejected, a team makes their service's own inbound request queue unbounded. During the next traffic spike, what actually happens to latency, memory and success rate, and what queue design would you use instead?
answer
- a queue absorbs bursts, not deficits
- depth divided by drain rate is waiting
- every queued request holds memory
- oldest request is likeliest already gone
- bound from latency budget, not heap size
basics
~20 sAn unbounded inbound queue converts an overload into unbounded latency plus a memory incident: the queue grows without limit, every request waits longer than its client's timeout, and the process eventually dies. A bounded, deadline-aware queue rejects immediately instead.
solid answer
~50 sQueueing does not create capacity; it only hides the shortfall until it becomes worse. If the queue holds 10,000 requests and the service drains 1,000 per second, the request at the head has already waited 10 seconds — with a 2-second client timeout, everything the service pulls off the front is dead on arrival, so it runs at 100% CPU with near-zero success rate. Meanwhile every queued request holds a buffer, so an unbounded queue is also an unbounded heap, and the process eventually OOMs or GC-thrashes. The design instead is: bound the queue, so exceeding it is an immediate 503 the client learns about in milliseconds rather than a timeout it learns about in ten seconds; check each request's deadline at dequeue time and drop expired ones without executing them; and under sustained overload consider serving newest-first, since the freshest request is the one whose caller may still be waiting. Facebook's `Fail at Scale` write-up describes exactly this pairing of adaptive LIFO with a CoDel-derived queue.
code
python · 23 linesimport time
from collections import deque
MAX_QUEUE = 500 # target_max_wait x service_rate
queue = deque()
def enqueue(req):
if len(queue) >= MAX_QUEUE:
return 503 # bounded: say no now, not in ten seconds
queue.append(req)
return 202
def next_request(now):
while queue:
req = queue.pop() # LIFO: freshest first under overload
if req["deadline"] > now:
return req # someone is probably still waiting
return None # everything queued was already dead
now = time.time()
enqueue({"id": 1, "deadline": now - 1})
enqueue({"id": 2, "deadline": now + 2})
print(next_request(now))go deeper
Know that putting requests in a queue does not create capacity, and that a queue with no size limit can grow until the process runs out of memory.
Be ready to do the arithmetic out loud: head-of-queue wait equals depth divided by drain rate, compare it to the client timeout, and explain why a bounded queue turns a timeout into a fast, useful rejection.
Demonstrate that you would make the queue deadline-aware, treat wait time rather than depth as the operational signal, and explain when you would switch service order under overload and what starvation that costs.
Own the policy: what maximum wait the platform sells to callers, whether queue discipline is a per-service choice or a shared default, and how deadlines are propagated so that dropping expired work is even possible.
## What a queue is actually for A queue in front of a service absorbs *bursts* — short periods where arrivals exceed the service rate, followed by periods where they do not. Averaged over the burst, arrivals must still be below capacity. If arrival rate exceeds service rate persistently, the queue is not absorbing anything; it is accumulating, and its depth grows without bound by construction. Queueing an overload does not solve the overload. It converts an immediately visible problem (errors) into two delayed and less tractable ones (unbounded latency, and unbounded memory). ## Latency: the arithmetic is brutal and simple Queue wait at the head is roughly depth divided by service rate. A queue holding 10,000 requests drained at 1,000 per second means the request being picked up right now has waited 10 seconds. Now overlay a client timeout — say 2 seconds. Anything that has waited more than 2 seconds has no caller left. In a FIFO queue, the request you dequeue is always the oldest, so once the depth exceeds timeout x service rate (here 2,000 entries), *every* request you serve is one that has already been abandoned. The service is fully busy and delivers zero goodput. From the outside it looks completely down, even though it is doing exactly as much work per second as it always did. This is also why queue depth alone is a poor health signal and queue *wait time* is a much better one: 5,000 entries is fine at 50,000 per second and catastrophic at 500 per second. ## Memory: the second incident Every queued request holds something — the parsed headers, the request body, a connection, a trace context, sometimes a pre-allocated response buffer. At a few kilobytes each, a queue that reaches a million entries is gigabytes of live heap. On a managed runtime you hit GC thrashing first: the collector runs constantly over a large live set, stealing the CPU that was supposed to drain the queue, which makes the queue grow faster. The end state is an OOM kill, which drops every queued request at once and — if the orchestrator restarts the instance — sends its share of traffic to the remaining instances, which are already overloaded. A memory-driven crash loop is a common way a soft overload becomes a hard outage. ## The design that works **Bound the queue.** Pick a bound from the latency you are willing to accept, not from available memory: `max_depth = target_max_wait x service_rate`. If you serve 1,000 per second and refuse to make anyone wait more than 500 ms, the bound is 500. Beyond it, reject. The client then finds out in milliseconds — early enough to fail over, try a different replica, or show a degraded page — instead of finding out at its timeout. **Make the queue deadline-aware.** If requests carry a deadline, check it when you dequeue and discard anything already expired without executing it. Dropping an expired request costs microseconds; running it costs its full service time. Under overload this alone can restore a large share of effective capacity. **Consider LIFO under overload.** FIFO is fair when the queue is short and pathological when it is long, because it systematically serves the requests most likely to be dead. Last-in-first-out serves the freshest request, whose caller is most likely still waiting. The cost is real: under sustained overload the entries at the bottom starve permanently. That is the tradeoff — LIFO gives up the promise of eventual service for a much higher share of useful completions. Facebook's `Fail at Scale` (ACM Queue, 2015) describes running FIFO normally and switching to LIFO adaptively when the queue is backed up, combined with a controlled-delay (CoDel) algorithm borrowed from network queue management that measures minimum queue delay over a window and starts dropping when it stays above a threshold. ```python req = queue.pop() # newest first while overloaded if req.deadline < now: continue # dead on arrival: drop, do not execute ``` ## What to monitor Queue wait time at the head, not just depth. The rate of deadline-expired drops — a sudden rise is the earliest honest signal that you are past capacity. And the rejection rate from the bound, which is your load-shedding rate, and should be an expected, graphed, non-alarming number during a spike rather than a surprise. ## The interview point The decision with a cost is how much waiting you are willing to sell. An unbounded queue is the choice to make every client wait arbitrarily long and never say no — which in practice means saying no to everyone, later, less usefully, with a memory incident attached. A bounded, deadline-aware queue is the choice to say no to some clients quickly so the rest get answers within their timeouts.
- How would you choose the maximum depth for a bounded request queue?Derive it from latency, not memory. Multiply the drain rate you actually sustain by the longest wait you are willing to impose: 1,000 per second and a 500 ms budget gives a bound of 500. Sizing it from free heap instead produces a queue that is technically survivable and operationally useless, because everything in it expires before it is served.
- Queue depth is the metric most teams graph. Why is queue wait time the better one?Depth has no meaning without the drain rate behind it — 5,000 entries is unremarkable at 50,000 per second and fatal at 500. Wait time is directly comparable to the client timeout, which is the number that decides whether work is useful, so it alerts correctly across traffic mixes and after capacity changes without retuning the threshold.
saying these in an interview costs you the question
- Thinks a queue adds capacity rather than deferring the shortfall
- Sizes the queue from available memory instead of latency
- Serves strict FIFO under sustained overload without question
- Never checks whether a dequeued request is still wanted
- Monitors queue depth without the drain rate beside it