In a multi-tenant job platform, one tenant enqueues two million exports at once; how would you keep that backlog from starving every other tenant?

## Why a global FIFO starves tenants A **multi-tenant** job platform serves many customers from one shared worker pool. With a single first-in, first-out queue, whoever enqueues first gets served first, and a single large backlog becomes everyone's problem. With illustrative numbers: one tenant enqueues 2,000,000 export jobs of about 30 seconds each, and the pool has 100 workers. That is 2,000,000 x 30 = 60,000,000 worker-seconds, divided by 100 workers = 600,000 seconds, about 6.9 days. Every other tenant's job submitted after that burst waits roughly a week. This is the **noisy neighbour** problem, and it is a scheduling-policy failure, not a capacity failure. ## Per-tenant queues and fair dispatch The fix is to schedule **tenants first, jobs second**: 1. Keep one logical queue per tenant, holding that tenant's jobs in its own order. 2. Keep a rotation of **active tenants**, those with queued work. 3. The dispatcher picks the next tenant from the rotation, takes some of its work, and moves on. Plain round-robin gives each tenant one job per turn. **Weighted** variants give paying or critical tenants more. **Deficit round-robin** also handles uneven job sizes: each tenant earns credit per turn and spends it by the estimated cost of the job it dispatches. ```pseudocode loop: tenant = activeTenants.next() tenant.deficit = min(maxDeficit, tenant.deficit + quantum * tenant.weight) while freeWorkers() > 0 and tenant.hasQueued() and tenant.running < tenant.cap: job = tenant.peek() if job.estimatedCost > tenant.deficit: break tenant.deficit -= job.estimatedCost dispatch(tenant.pop()) tenant.running += 1 if not tenant.hasQueued(): tenant.deficit = 0 activeTenants.remove(tenant) ``` Dispatch cost grows with the number of **active tenants**, not the number of queued jobs, so a two-million-job backlog does not slow the dispatcher. The deficit ceiling stops a capped tenant from building up unlimited credit and then bursting. ## Concurrency caps: hard versus work-conserving Fair ordering alone is not enough: if the big tenant's jobs are long, it can still end up holding most workers once they are running. A **per-tenant concurrency cap** limits how many of its jobs run at once. | Cap style | Behaviour | Trade-off | |---|---|---| | Hard cap | never more than N running | strong isolation, but workers sit idle while the capped tenant still has work | | Work-conserving cap | the cap applies only while others are waiting | uses idle capacity, but a newly arriving tenant must wait for borrowed workers to finish | | Cap plus reserved share | each tier keeps a guaranteed floor, the rest is shared | predictable for premium tenants, more configuration to own | The borrowing delay in the work-conserving row is why long-running job types need **preemption points** or shorter job units: a tenant can only reclaim its share as fast as borrowed jobs finish. ## Fair by what? - **Job count** is simple but unfair when sizes vary: one two-hour render and one two-second export count as equal turns. - **Worker-seconds consumed**, scaled by weight, reflects the real share of the pool; it needs estimated or measured job cost. - **Hierarchical fairness** shares first between organisations and then between their projects, so a customer cannot gain share by splitting into many sub-accounts. - **Within one tenant**, the tenant's own priorities decide order, so fairness between tenants does not dictate which of a tenant's jobs runs first. ## Isolation beyond the dispatcher - **Admission quotas** at enqueue time limit how fast or how much one tenant can submit, turning a burst into back-pressure at the API instead of a hidden week-long queue. - **Separate worker pools** for job types with very different profiles, such as renders versus exports, stop long jobs from occupying the slots short jobs need. - **Shared downstream limits** matter too: fair dispatch is useless if every export hammers the same database, so per-tenant limits may be needed at those dependencies as well. - **Retries count toward the tenant's share**, otherwise a tenant with a failing integration can take extra capacity by retrying. ## Measuring fairness - Track **queue wait time per tenant**, especially the high percentiles. A good global average can hide one tenant waiting hours. - Track **running jobs and worker-seconds per tenant** against their weight or cap. - Alert when any tenant's oldest queued job exceeds its target, not only when the whole queue is deep. - Review weights and caps with product owners: they encode a business decision about who gets served first under contention.