You're designing a placement/bin-packing strategy for a shared fleet running thousands of heterogeneous workloads (mixed CPU-bound, memory-bound, and I/O-bound; mixed criticality). What are the key design decisions you'd need to make to maximize density without creating systemic contention risk?

## Five interacting layers At the scale of thousands of heterogeneous workloads, compute resource consolidation stops being a simple 'pack workloads onto fewer machines' exercise and becomes a systems design problem with several interacting layers: 1. **Classification** 2. **Placement** 3. **Governance** 4. **Feedback** 5. **Degradation policy** Getting each layer right is what separates a fleet that steadily improves utilization from one that accumulates unpredictable contention incidents as it scales. ## Classification is the foundation The foundation is workload classification. Every workload needs at least two independent labels driving how it can be packed: - a **resource profile** — is it CPU-bound, memory-bound, I/O-bound, or some mix, and how bursty/variable is its usage over time; - a **criticality/SLA tier** — can it tolerate throttling and eviction, or does it need guaranteed capacity. These labels should come from real historical telemetry — CPU/memory/IO usage percentiles over representative windows, not developer guesses at deploy time, because self-reported resource requests are notoriously inaccurate (both over- and under-stated) and drift as code changes. Classification is what makes multi-dimensional bin-packing possible: it lets the placement layer distinguish 'this workload has spare CPU most of the time' from 'this workload is quietly memory-hungry,' which raw declared requests alone don't capture. ## The placement algorithm The second layer is the placement algorithm itself. A pure bin-packing heuristic applied to a single resource dimension will happily create nodes that are CPU-full but memory-starved, or vice versa, so at fleet scale the scheduler needs to score candidate nodes across multiple resource dimensions simultaneously and weight them according to which dimension is actually scarce in that part of the fleet. On top of the raw packing objective, the classification labels from layer one feed affinity and anti-affinity rules: - pack complementary-profile workloads together to interleave their resource usage; - enforce anti-affinity between workloads whose criticality tiers or compliance scopes shouldn't share a fault domain, and between workloads whose peak-usage windows are known to correlate. This is also where **node pool segmentation** becomes practical at scale: rather than expressing every constraint as a per-workload rule, group nodes into pools by policy (e.g., a 'guaranteed' pool for top-tier SLA workloads with generous headroom, a 'burstable' pool for elastic/best-effort work packed much more tightly) and let the scheduler place workloads into the pool matching their tier. ## Governance makes placement hold The third layer, governance, is what makes the placement decision actually hold at runtime rather than just on paper. Every workload needs: - enforced resource requests and limits (cgroups underneath, Kubernetes requests/limits or ECS task reservations at the orchestration layer) consistent with its classification; - plus a priority/QoS class that determines eviction and throttling order when a node comes under pressure despite correct placement — because even good bin-packing doesn't eliminate the tail case of an unexpected burst. Without enforced governance, the classification and placement work is advisory only, and a single misbehaving workload can still degrade the whole node regardless of how carefully it was placed. ## Continuous feedback The fourth layer is continuous feedback. Workload resource profiles and criticality aren't static — a service's traffic pattern shifts, a new feature changes its memory footprint, a previously best-effort batch job becomes business-critical. A fleet-scale consolidation strategy needs a control loop: - ongoing collection of actual usage versus declared requests/limits; - automated or human-reviewed re-tuning of both the classification labels and the request/limit values; - ideally periodic re-bin-packing (descheduling and rescheduling) to correct placement decisions that have gone stale as the fleet's workload mix has shifted. Treating the initial placement as permanent is a common failure mode — utilization degrades slowly and invisibly as workloads drift away from the assumptions that justified their original placement. ## Explicit degradation policy The fifth and often underweighted layer is explicit degradation policy: deciding, in advance, what happens when consolidation's assumptions are violated anyway — a node genuinely runs hot despite correct classification and placement. This means defining: - **eviction order** — which QoS/priority tier gets evicted first under memory pressure; - **throttling behavior** — which workloads absorb CPU contention first; - **alerting thresholds** that distinguish 'expected, bounded degradation of a best-effort workload' from 'an SLA-tier workload is being impacted, which should never happen and needs investigation.' Without this explicit policy, contention incidents get resolved ad hoc during an outage, which is both slower and more likely to make the wrong trade-off under pressure. ## Consolidation as a control loop The overarching principle tying these five layers together is that fleet-scale consolidation is a control loop, not a one-time placement decision: 1. Classify based on real data. 2. Place using multi-dimensional bin-packing constrained by that classification. 3. Govern so placement decisions hold under load. 4. Feed usage data back to keep classification accurate. 5. Define degradation behavior for when the system is oversubscribed anyway. This is essentially the operating model behind large-scale production schedulers — Google's Borg/Omega lineage, which pioneered priority-based eviction and usage-based reclamation of unused reservations, and Kubernetes clusters run with node pools segmented by workload class are both concrete, real implementations of this layered approach, and both explicitly treat 'declared request equals actual guarantee equals correct placement forever' as a false assumption that has to be continuously corrected.