Describe the extension points of the Kubernetes scheduling framework and how you would customise placement without forking kube-scheduler.

## Why the framework exists Before Kubernetes 1.15-ish, changing placement meant either a scheduler "policy" file with a fixed vocabulary of predicates and priorities, or an HTTP **Extender** the scheduler called out to — a network hop in the middle of a hot loop, with no access to scheduler state. The **scheduling framework** replaced that with an in-process plugin architecture: every built-in behaviour is itself a plugin registered at one or more extension points, so third-party logic is a first-class citizen rather than a bolt-on. ## The extension points, in order **Queue and entry** - **PreEnqueue** — decides whether a pod may enter the active queue at all. `SchedulingGates` lives here: a pod with `spec.schedulingGates` set is held out of scheduling entirely until a controller removes the gates. This is how you implement "do not schedule until quota is approved / peers exist". - **QueueSort** — exactly **one** plugin may be enabled; it defines the ordering of pending pods. The default sorts by pod priority descending, then by the timestamp at which the pod became unschedulable. **Scheduling cycle (one pod at a time, serial)** - **PreFilter** — pod-level checks and precomputation whose result is shared with later points via a `CycleState` object. It may reject the pod outright ("no node can ever satisfy this"), saving a full node sweep. - **Filter** — called per node; returns feasible/infeasible with a reason string. This is where `NodeResourcesFit`, `NodeAffinity`, `TaintToleration`, `NodePorts`, `VolumeBinding`, required `PodTopologySpread` and `InterPodAffinity` live. Runs in parallel across nodes internally, and stops early once `percentageOfNodesToScore` worth of feasible nodes are found. - **PostFilter** — invoked **only** if the feasible set is empty. `DefaultPreemption` implements victim selection here. A PostFilter plugin can also simply produce a better failure message. - **PreScore** — precompute for scoring (e.g. build the topology map once instead of per node). - **Score** — per feasible node, returns an integer; **NormalizeScore** rescales a plugin's raw range into 0–100; the framework multiplies by the plugin **weight** and sums. Highest total wins, random tiebreak among equals. - **Reserve / Unreserve** — the plugin records that the pod is taking something (the volume binder reserves PVC bindings here). `Unreserve` is the rollback path if anything later fails, which makes Reserve the place to hold *stateful* claims rather than doing it in Score. - **Permit** — the most powerful hook: it can **approve**, **deny**, or return **wait** with a timeout. A waiting pod holds its reservation while the plugin decides. Coscheduling (gang scheduling for MPI/Spark jobs) is built here: each pod of the gang waits at Permit until enough peers have arrived, then all are released together — or all time out and are unreserved. **Binding cycle (asynchronous, concurrent with the next pod's scheduling cycle)** - **WaitOnPermit** → **PreBind** (do the real external work, e.g. finalise volume binding — failures here trigger Unreserve) → **Bind** (write the Binding; the first plugin to handle it wins, so a custom Bind plugin can integrate an external placement system) → **PostBind** (cleanup/notification, cannot fail the scheduling). ## Configuring without writing code Most "custom scheduling" needs are met by a `KubeSchedulerConfiguration`: ```yaml apiVersion: kubescheduler.config.k8s.io/v1 kind: KubeSchedulerConfiguration profiles: - schedulerName: default-scheduler plugins: score: enabled: - name: NodeResourcesBalancedAllocation weight: 5 disabled: - name: ImageLocality pluginConfig: - name: PodTopologySpread args: defaultingType: List ``` A **profile** is a named bundle of plugin enablement, weights and args. Several profiles can live in one scheduler process; a pod picks one with `spec.schedulerName`. That is how a cluster offers, say, a spread-oriented default and a bin-packing profile for batch, without running two binaries. ## Writing a plugin An out-of-tree plugin implements the Go interfaces for the points it cares about (`FilterPlugin`, `ScorePlugin`, …), registers itself in a `main.go` built on `k8s.io/kubernetes/cmd/kube-scheduler/app`, and is deployed as *your* scheduler image — either replacing the default scheduler or running as a second deployment with a distinct `schedulerName`. Practical constraints: the plugin runs in the hot path, so it must be fast and use informer caches rather than API calls; it must be deterministic enough to be debuggable; and it takes on the framework's Go version and API compatibility. The upstream `kubernetes-sigs/scheduler-plugins` repository packages the common ones (coscheduling, capacity scheduling, node-resource-topology, trimaran load-aware scoring), which is usually a better starting point than a blank file. ## Choosing the right level Running a **second scheduler** is fine but puts you in charge of the split-brain risk: two schedulers making decisions against independent caches can both place a pod's worth of resources on the same node, and only kubelet admission catches it. The old **Extender** webhook still exists for filter/prioritise/bind, but it costs a network round trip per pod and cannot see scheduler state — reach for it only to integrate an external system you cannot link into Go. And before any of this: most placement intent is already expressible with node affinity, taints, topology spread constraints and resource requests. Custom plugins earn their keep for genuinely different *semantics* — gang scheduling, quota-aware placement across queues, real-utilisation-aware scoring — not for preferences the built-ins already model.