How would you design YARN queues for a cluster shared by teams with different SLAs?

## Start from promises, not from teams The common failure is one queue per team. It looks fair and it fails: each team's queue mixes a deadline-bound pipeline with somebody's exploratory notebook, so the scheduler cannot tell what matters, and fifteen small guarantees fragment the cluster so nothing large fits anywhere. Design instead around the *promise* the platform makes, then place teams inside that structure. A layout that survives contact with reality typically has three classes under `root`: - **Production / SLA** — scheduled pipelines with deadlines. Guaranteed capacity sized to measured peak concurrent demand, `maximum-capacity` only modestly above it, FIFO ordering inside (deterministic, and these jobs are large). - **Interactive / ad-hoc** — notebooks, SQL, exploration. Moderate guarantee so responsiveness is protected, `fair` ordering inside so a 30-second query never waits behind a 3-hour one, tight per-user and per-application concurrency limits. - **Best-effort / batch** — backfills, retraining, experiments. Small guarantee, ceiling near 100 so it absorbs everything idle, and the first queue preempted. Sub-queues per team belong *inside* a class, where teams share that class's guarantee, rather than at the top. ## Sizing the guarantees A guarantee should reflect measured peak concurrent demand, not a team's headcount or its budget contribution. Look at the pending-container time per queue over a few weeks: a queue whose applications rarely wait is over-provisioned, one that waits constantly is under-provisioned or has a concurrency problem rather than a capacity one. Because guarantees must sum to 100 under `root`, every increase is a decision to take capacity from somebody — which is exactly the conversation the design should force into the open rather than resolving silently at 3am. ## Elasticity is the whole point of sharing If every queue's `maximum-capacity` equals its `capacity`, you have built several small static clusters that happen to share hardware, and utilisation collapses. If every ceiling is 100, utilisation is excellent but guarantees are honoured slowly, because a borrowed container is only released when it finishes. The design decision is where each queue sits on that line: high ceilings for interruptible work, low ceilings for work whose containers are long-lived and expensive to lose. ## Preemption makes a guarantee real — at a price Without preemption a "guarantee" is a promise of eventual capacity, delivered as borrowed containers drain. With an eight-hour borrower, that is an eight-hour SLA breach. The Capacity Scheduler's scheduler monitor with a proportional preemption policy reclaims a starved queue's share by killing borrowed containers. The cost is discarded work, so calibrate rather than switch it on globally: use a grace period, prefer to preempt the best-effort class, and exempt workloads whose containers hold expensive state. Treat a persistently high preemption rate as evidence the guarantees are wrong, not as steady-state operation. ## Concurrency control is as important as capacity Most "the cluster is full" incidents are concurrency incidents. Four knobs matter: `maximum-applications` per queue caps how many can be pending or running at once; `maximum-am-resource-percent` (0.1 by default) bounds how much of a queue can be spent on coordinators, which prevents the deadlock where every resource is a coordinator waiting for workers; `user-limit-factor` stops a single user from consuming the queue's elastic headroom; and `minimum-user-limit-percent` divides a queue among active users. An interactive queue without these is one runaway notebook away from an outage. ## Placement, access and accountability Queue mappings route users and groups to queues automatically so nobody has to pass a queue name, and per-queue submit and administer ACLs stop teams landing in the production queue. Make placement automatic and auditable — a manual queue name in a job config is a bug waiting to happen. Pair this with per-queue chargeback reporting: teams argue for larger guarantees indefinitely until the cost of that guarantee is visible to them. ## Also decide what YARN cannot do for you A scheduler shares CPU and memory. It does not isolate disk or network bandwidth well, so a queue can be within its memory guarantee and still degrade neighbours through shuffle traffic or local disk saturation. It does not protect the storage layer from a queue hammering it. And it does not size the cluster — if every queue is starved, no layout fixes it. Know which problems are queue-design problems and which are capacity or workload problems, because reshuffling percentages to solve a capacity shortage burns credibility. ## Treat it as a living hypothesis Publish the layout and the reasoning, instrument pending time, preemption rate and utilisation per queue, and revisit quarterly. The strongest answer here is not a specific set of percentages — it is a design that states its assumptions, measures whether they hold, and has a defined path for a team to request a change.