A Kubernetes node goes `NotReady`, and after roughly five minutes its pods are deleted and recreated elsewhere. Explain the mechanism that does this, and how it differs from a kubelet evicting pods because the node is short on memory.

## The heartbeat Each kubelet reports liveness to the API server through a **Lease** object in `kube-node-lease`, renewed every few seconds. The **node controller** inside kube-controller-manager watches those leases. If a node's lease stops being renewed for `node-monitor-grace-period` (40s in typical upstream configuration), the controller sets the node's `Ready` condition to `Unknown` — this is what `kubectl get nodes` shows as `NotReady`. The node may be genuinely dead (hardware failure, kernel panic), or merely **partitioned** — still running every container, just unable to talk to the control plane. From the control plane's viewpoint these are indistinguishable, and that ambiguity drives everything that follows. ## Taint-based eviction On marking the node unhealthy, the node controller applies a taint: - `node.kubernetes.io/not-ready:NoExecute` when `Ready=False`, - `node.kubernetes.io/unreachable:NoExecute` when `Ready=Unknown`. A **taint** repels pods; a **toleration** in a pod spec lets it stay. Taint effects: `NoSchedule` (do not place new pods), `PreferNoSchedule` (avoid if possible), and `NoExecute` (also remove pods **already running** that do not tolerate it). The component that acts on `NoExecute` is the **taint manager**. For each running pod on the tainted node it checks the pod's tolerations: - no matching toleration → delete immediately; - matching toleration with no `tolerationSeconds` → stay forever; - matching toleration with `tolerationSeconds: N` → delete after N seconds. ## Where the five minutes comes from The API server runs the **DefaultTolerationSeconds** admission plugin. Every pod created without explicit tolerations for these two taints gets: ```yaml tolerations: - key: node.kubernetes.io/not-ready operator: Exists effect: NoExecute tolerationSeconds: 300 - key: node.kubernetes.io/unreachable operator: Exists effect: NoExecute tolerationSeconds: 300 ``` Hence the familiar ~5-minute delay before pods on a dead node are recreated (roughly 40s to notice plus 300s of toleration). The value is tunable per pod, or cluster-wide via the API server's `--default-not-ready-toleration-seconds` / `--default-unreachable-toleration-seconds`. Lowering it recovers faster but reacts to transient network blips; raising it tolerates flaky networks at the cost of longer outages. DaemonSet pods are given tolerations without `tolerationSeconds` for these taints, so they stay put — they belong to the node, and there is nowhere to move them. ## Deleted, not stopped The taint manager deletes the **pod object**. If the node is unreachable, the kubelet never receives the delete, so the containers may keep running — the pod becomes an orphan still holding whatever the workload holds. Kubernetes calls the deleted-but-possibly-alive state exactly why StatefulSet replicas are not automatically recreated for unreachable nodes: recreating a pod with the same identity while the original might still be writing risks split-brain. Forcing it (`kubectl delete pod --force --grace-period=0`) tells the API server to drop the record without confirmation and is safe only once you have genuinely fenced the node (powered it off, detached its volumes). ## Contrast with node-pressure eviction Node-pressure eviction is a **local, resource-driven** action by the kubelet: a signal like `memory.available` breaches a threshold, the kubelet ranks pods by usage-versus-request and priority, and kills enough of them to recover. The node stays `Ready`; only some pods die; the evicted pods leave `Evicted` tombstones behind. Taint-based eviction is a **remote, health-driven** action by the control plane: the node's health, not its resource usage, is the trigger; *all* non-tolerating pods go, regardless of QoS; the pods are deleted outright rather than left as `Evicted` records. Both ignore PodDisruptionBudgets — PDBs constrain only API-initiated eviction through the `/eviction` subresource, which is what `kubectl drain` and cluster autoscalers use. ## Operator-applied NoExecute taints The same machinery serves deliberate operations: `kubectl taint node n1 maintenance=true:NoExecute` drains non-tolerating pods off a node. Pressure conditions, by contrast, generate `NoSchedule` taints (`node.kubernetes.io/memory-pressure`, `disk-pressure`) — they stop new placements while the kubelet handles removals locally. ## Interview framing Name the actors (node controller → taint → taint manager), explain the 300-second default toleration as the source of the five minutes, note that pods are *deleted* while containers may still run on a partitioned node, and contrast trigger/actor/victim-selection with kubelet node-pressure eviction.