A service uses two separate queues — high-priority and low-priority — for order processing, both backed by the same consumer pool and the same downstream database. During an incident, a bug causes low-priority messages to be reprocessed in a tight retry loop, and soon high-priority order confirmations start timing out too. What went wrong with the isolation design, and how would you fix it?

## What actually went wrong This scenario illustrates a subtle but common mistake: treating 'separate queues' as equivalent to 'isolated priorities,' when in fact the queues are only the first hop. A message's priority is meant to protect end-to-end processing time, but that protection only holds if every shared resource downstream of the queue is also partitioned — or at least rate-limited — per priority tier. In the scenario described, both the high-priority and low-priority queues fed the same consumer pool and the same backend database. Splitting the queues successfully controlled which messages a consumer picks up first, but once a message is dequeued, it competes for the exact same finite resources (consumer threads, database connection-pool slots, database CPU/IO) as messages from the other tier. A retry storm on the low-priority side — say, a bug causes a transient error to be treated as retryable and messages are reprocessed in a tight loop with no backoff — floods that shared pool of consumer threads and database connections. Even though the priority queue policy still correctly hands high-priority messages to a consumer thread first when one is free, if no threads are free (because they're all stuck retrying low-priority work) or if the database itself is saturated (because retries are hammering it), high-priority processing degrades or times out too, defeating the entire point of the priority separation. ## The mechanism to understand This is fundamentally a resource-isolation failure layered underneath a correctly designed message-routing decision. The mechanism to understand is: **priority queues control order of dequeue, not resource consumption after dequeue.** Anything that happens after a message leaves the queue — database writes, downstream API calls, retries — is a shared-resource problem, and shared resources are exactly what allow low-priority work to degrade high-priority work despite correct queue-level separation. This is analogous to **noisy-neighbor** problems in multi-tenant systems generally: putting two tenants' work in different queues doesn't protect one tenant from the other if they still share a database connection pool underneath. ## Fix one — extend the isolation boundary past the queue The fix has two parts. First, extend the isolation boundary past the queue and into every resource the message touches: - **dedicated consumer pools per priority tier**, so a broken or overloaded low-priority pool cannot consume threads that high-priority processing needs; - and where the downstream dependency allows it, **dedicated connection pools, rate limits, or even physically separate database read replicas or shards** per tier for the highest-value traffic. This doesn't have to mean fully separate infrastructure for every tier — for many systems, a modest reserved connection-pool allocation per tier (e.g., high-priority consumers get their own pool of 20 DB connections that low-priority consumers can never use, even if idle) is enough to guarantee a floor of throughput for the priority tier without doubling infrastructure cost. ## Fix two — let the low-priority path degrade gracefully Second, and just as important, the low-priority path itself needs safeguards so a bug there degrades gracefully instead of consuming unbounded shared capacity. Concretely: - **retry policies** with a maximum attempt count and exponential backoff (never infinite immediate retry); - **a dead-letter queue** that a message is moved to after exceeding the retry limit so it stops consuming processing capacity and instead waits for manual/automated remediation; - **a circuit breaker** around the downstream call so that once error rates spike, the consumer stops hammering the failing dependency instead of retrying in a tight loop. These are general resilience patterns, but they're doubly important on the low-priority path specifically, because a failure there is easy to overlook operationally — nobody's paging on low-priority queue health the way they are on high-priority — until it has already consumed enough shared capacity to affect the tier that actually matters. ## Where it shows up A concrete real-world parallel: an e-commerce platform split order-confirmation emails (high-priority) from promotional/marketing emails (low-priority) into separate SQS queues, but both were consumed by the same Lambda concurrency pool with a shared downstream email-provider API rate limit. A marketing send with a malformed template caused every message to fail and retry, and because SQS's default visibility-timeout-based retry has no built-in backoff cap without explicit configuration, the retries consumed the account's entire Lambda concurrency limit and the email provider's rate limit within minutes — delaying order confirmations by nearly an hour until an engineer manually purged the marketing queue. The postmortem fix was exactly the pattern above: separate Lambda reserved-concurrency allocations per queue, a dead-letter queue with a max receive count on the marketing queue, and a per-queue rate limit against the email provider.