Scaling and Scheduling

Inter-Pod rules are evaluated per candidate node against every already-placed matching Pod and its node's topology label, so cost scales roughly with nodes times matching Pods, unlike node affinity's single label comparison. Limit selectors, prefer hostname topology, and prefer soft rules.

Pods must fit whole on one node, so free capacity summed across nodes means nothing. Many small leftover gaps, or free CPU stuck next to exhausted memory, strand capacity. Reduce it by repacking pods, standardising request shapes and scoring toward fuller nodes.

CPU-based horizontal scaling measures usage divided by the request. VPA moves the request, so the same real load changes the measured utilization and makes the horizontal controller scale in or out for no workload reason. The two form a feedback loop. Split them: VPA on memory only, horizontal scaling on CPU or on custom metrics.

Taint the GPU nodes (nvidia.com/gpu:NoSchedule) to keep CPU-only pods off, and label them so GPU pods can select them. Enabling the ExtendedResourceToleration admission plugin adds the matching toleration to any pod that requests nvidia.com/gpu.

The behavior block. behavior.scaleDown.stabilizationWindowSeconds (default 300) makes the controller take the highest recommendation from the last window before shrinking; the scaleUp window (default 0) takes the lowest. Policies under each direction cap how many pods or what percentage may change per period. Widen the scale-down window and add a percent/pod policy to damp flapping.

Use the autoscaling/v2 metric types Pods, Object or External instead of Resource. Those are served by custom.metrics.k8s.io and external.metrics.k8s.io, which core Kubernetes does not implement — you install an adapter (Prometheus Adapter, KEDA, a cloud provider's) that translates a query into that API. Use External with AverageValue for queue depth so the target means backlog per pod.

The delay is a chain: up to one pollingInterval before KEDA notices, then scheduling, start-up and readiness, then the consumer joining its group. Measure each stage; for a tight budget, keep minReplicaCount at 1 instead.

Consolidation deletes or replaces nodes whose pods fit elsewhere more cheaply, and drift replaces nodes that no longer match their NodePool or NodeClass. Limit both with consolidateAfter, disruption budgets scoped by reason and schedule, PDBs, and the do-not-disrupt annotation.

Find which PDB has disruptionsAllowed = 0 with kubectl get pdb -A, then check why: replicas equal to minAvailable, maxUnavailable 0, pods failing readiness so they never count as healthy, a selector spanning several workloads, or overlapping PDBs. Fix the cause — scale up, correct the budget, fix readiness — rather than deleting the PDB.

They are reserved classes for infrastructure: system-cluster-critical (2000000000) for control-plane-level add-ons like CoreDNS, and system-node-critical (2000001000, the highest) for per-node agents like the CNI and kube-proxy. Both sit above the one-billion user ceiling. Never put application workloads in them.

Guarantee an on-demand floor that can serve peak alone, let only the surplus replicas use spot, spread both across zones, and cut shutdown time to fit the reclaim notice, because a disruption budget cannot stop a reclaim.

The node lifecycle controller taints nodes from their reported conditions. not-ready and unreachable are NoExecute, so pods are evicted once their injected 300-second toleration expires; pressure taints such as disk-pressure and memory-pressure are NoSchedule and only keep new pods away. DaemonSets tolerate them so node agents keep running.

Most likely: the constraint is ScheduleAnyway so it only scores; or the domains were unequal at placement time (a zone was full or unavailable) and nothing rebalances afterwards; or nodes lack the zone label and are excluded; or the labelSelector is counting the wrong pod set. Check the constraint, the node labels, and the actual per-zone counts.

Anti-affinity is binary per domain — required anti-affinity allows at most one matching pod per domain, which caps replicas at the number of domains. Spread constraints express a degree of evenness via maxSkew, so many pods per domain are fine. Use anti-affinity when you truly need at most one, or to express repulsion from different pods.

Not yet: the scheduler budgets requests, and at 95% requested the cluster cannot even absorb one node loss. Reduce requests first, repack, keep an N+1 ceiling near 89% requested, and only then return hardware.

Set scale-down aggressiveness per pool, not globally. Allow vetoes only as reviewed exceptions: budgets that allow a disruption, and time-limited pins on a tainted pool. Report stranded node-hours per team so each veto's cost is visible.

Let the workload mix set the spot share, not the discount: only interruption-tolerant work goes there, critical services keep an on-demand floor that serves peak alone, spot pools are diversified, and admission policy enforces who may tolerate them.

kube-scheduler decides only for unbound pods; once bound, a pod stays put. The descheduler is a separate add-on that evicts selected pods through the Eviction API so their controllers recreate them and the scheduler places them again.

A ScaledObject scales a long-running Deployment through a generated HPA. A ScaledJob creates Kubernetes Jobs that run to completion. Use a ScaledJob for long tasks that scale-in must never interrupt.

Cluster Autoscaler simulates the Pending pods against a template node for each node group, then an expander picks among the groups that fit. The default expander, least-waste, picks the group that leaves the least CPU and memory unused.

Placeholder pods with a very low PriorityClass reserve spare capacity. Displaced pods preempt them and start at once, and the now-Pending placeholders make the Cluster Autoscaler add a node to restore the headroom.

Dynamic Resource Allocation replaces an anonymous device count with published device inventories (ResourceSlices) and claims (ResourceClaims) that select devices by attributes using CEL. The scheduler picks specific devices, and claims can be shared or given fallbacks. It is GA since Kubernetes 1.34.

kube-scheduler is a plugin host with ordered extension points: QueueSort, PreFilter, Filter, PostFilter, PreScore, Score, Reserve, Permit, then PreBind, Bind, PostBind. You customise by writing a KubeSchedulerConfiguration profile that enables, disables or reweights plugins, or by compiling in your own plugin and selecting it via spec.schedulerName.

Use soft podAffinity from the service to the cache at zone topology to cut cross-zone latency and cost, plus podAntiAffinity on the service's own label for spreading. Keep both soft — hard co-location plus hard spreading over-constrains the scheduler and produces Pending Pods.

Cluster Autoscaler resizes pre-defined, fixed-shape node groups. Karpenter reads Pending Pods and launches individually chosen instances from a broad type list, then consolidates. Choose it for heterogeneous workloads, faster provisioning and better bin-packing; you give up predictability, a cloud-agnostic component, and stable long-lived nodes.

Use few, widely spaced tiers you can explain during an incident — roughly platform, production, internal, batch, preemptible — set the global default low, grant preemption rights only where start time must be bounded, and control who may use each tier with scoped quotas. Alert on preemption rate as a capacity signal.

LeastAllocated favours the emptiest node, spreading pods for headroom and blast-radius safety. MostAllocated packs pods onto the fullest node that still fits, leaving nodes empty enough to be removed and cutting cost. Choose spread for latency-sensitive services, packing for batch on autoscaled or spot capacity.

Taint only where sharing genuinely fails: scarce or expensive hardware, licensing, compliance boundaries, and disruption-prone nodes like spot. Own the taint keys centrally, enforce who may tolerate them at admission, and accept the cost — every exclusive pool needs its own spare capacity and headroom, so partitioned clusters are more expensive than shared ones.

Default to soft zone and hostname spreading for everything (ScheduleAnyway, small maxSkew) via the scheduler's default constraints, and require an explicit opt-in for hard DoNotSchedule constraints, reserved for quorum and compliance workloads. Hard defaults strand pods during zone outages and scale-ups; soft defaults cost some imbalance but never block capacity.