kube-scheduler's NodeResourcesFit plugin scores nodes with a LeastAllocated strategy by default, and can be configured to MostAllocated. What does each choice do to a cluster, and how would you decide between them?

## What the strategies actually compute The filter phase has already reduced the cluster to nodes where the pod *fits*. Scoring only decides which of those is preferred. `NodeResourcesFit` computes, per resource, a ratio of **requested** to **allocatable** on the node *including* the incoming pod, then converts it to a 0–100 score: - `LeastAllocated`: score rises as the requested fraction falls — the emptiest feasible node wins. - `MostAllocated`: score rises as the requested fraction rises — the fullest feasible node wins. - `RequestedToCapacityRatio`: you supply a shape (a set of utilisation→score points) and per-resource weights, so you can express "prefer nodes between 60% and 80% utilised" or "pack GPUs hard but spread CPU". Remember the currency is **requests**, not observed usage. Bin-packing by requests packs *reservations*; if requests are wildly larger than reality you pack empty air, and if they are smaller than reality you pack contention. Any packing strategy is downstream of request accuracy — which is why VPA recommendations, or at least a periodic right-sizing pass, are a prerequisite rather than a nice-to-have. ## The case for spreading (LeastAllocated) - **Burst headroom.** A container may exceed its CPU request whenever the node has spare cycles; CFS only throttles at the *limit*. On an emptier node, bursts are actually servable, so p99 latency improves. Memory is harsher: exceeding a limit is an OOM kill, and on a packed node a memory-pressure eviction can take neighbours with it. - **Blast radius.** If each service has 3 replicas across 20 lightly loaded nodes, losing a node costs a fraction of one service. On a packed cluster the same node might hold every replica's worth of several services. - **Simpler operations.** Draining a node for an upgrade moves fewer pods, so PodDisruptionBudgets are easier to satisfy. The cost is money and fragmentation: nodes hover at 40% requested, none is empty enough for the autoscaler to remove, and you pay for the gaps. ## The case for packing (MostAllocated) - **The Cluster Autoscaler removes empty nodes, not half-empty ones.** It scales down a node when its pods can be rescheduled elsewhere and utilisation is below a threshold for a period. Spreading actively fights this: every node holds a few pods, so none qualifies. Packing concentrates load and *creates* removable nodes. On an autoscaled cluster the scoring strategy is one of the few direct levers on node count, and node count is the bill. - **Large-pod feasibility.** Fragmentation is the enemy of a pod that needs 8 CPU: twenty nodes with 4 CPU free each cannot host it. Packing preserves contiguous space. - **Spot/preemptible fleets** are cheaper per node but expected to churn; concentrating disposable batch work on few nodes matches the model. The cost is interference (CPU cache, memory bandwidth, disk and network I/O are not isolated by requests), larger per-node blast radius, and heavier drains. ## Deciding The decision is not global — it is per workload class, which is why **profiles** exist: ```yaml profiles: - schedulerName: default-scheduler # services: spread - schedulerName: bin-packing # batch: pack pluginConfig: - name: NodeResourcesFit args: scoringStrategy: type: MostAllocated ``` Workloads opt in with `spec.schedulerName`. A useful default split: customer-facing services stay on the spreading default and rely on **topology spread constraints** for availability; batch, CI and analytics use the packing profile on their own node pool. Other knobs that belong in this conversation: - **`NodeResourcesBalancedAllocation`** prefers nodes where CPU and memory end up *evenly* consumed, avoiding the classic stranded-resource shape (a node with 12 GB free but no CPU). Keep it enabled under either strategy; it attacks fragmentation from a different angle. - **Topology spread constraints** are the correct tool for availability, not the scoring strategy. Packing plus a `maxSkew: 1` spread over `topology.kubernetes.io/zone` gives you cost *and* survivability — the spread constraint filters or scores for distribution while `MostAllocated` chooses densely *within* the allowed set. - **Node shape matters more than the strategy.** Very large nodes amplify blast radius and make packing decisions coarse; very small ones make DaemonSet overhead (one copy per node) a real tax. Sizing the pool is often the higher-leverage decision. - **PodDisruptionBudgets** determine whether the autoscaler can actually act on the empty-ish nodes packing creates. A batch tier with restrictive PDBs blocks scale-down and negates the saving. ## How to validate the choice Instrument before and after: node count over time, cluster-wide requested-versus-allocatable ratio, p99 latency and CPU throttling for the service tier, OOM-kill and eviction rates, and autoscaler scale-down event counts. If packing does not reduce node count, the binding constraint is elsewhere — usually PDBs, local storage, or pods that cannot be rescheduled — and you have taken on interference risk for nothing. ## The honest summary There is no universally right answer, which is exactly why it is configurable. Spreading buys latency stability and small blast radius with money; packing buys money with interference risk and larger failure domains. On fixed on-prem capacity the money argument mostly evaporates and spreading usually wins; on an elastic cloud cluster with accurate requests and disposable batch work, packing pays for itself.