In a Kubernetes pod spec, what does the `tolerationSeconds` field on a toleration do, and how does it interact with a taint whose effect is NoExecute?

## The NoExecute effect The `NoSchedule` and `PreferNoSchedule` taint effects are pure scheduling-time constructs. `NoExecute` is different: besides filtering the node out for pending pods, it activates a controller loop — commonly called the *taint manager*, part of the node lifecycle controller inside `kube-controller-manager` — that watches nodes and pods and removes pods that are running on a node carrying an un-tolerated `NoExecute` taint. That is what makes taints usable as a live quarantine mechanism instead of only a placement rule. ## What tolerationSeconds adds Without `tolerationSeconds`, a toleration is a permanent exemption: the pod can sit on the tainted node indefinitely. `tolerationSeconds` converts it into a *grace period* — "I can live with this taint, but only for so long". The semantics: - The clock starts when the matching taint is **added** to the node, not when the pod started. - When the timer expires the taint manager deletes the pod. The workload controller behind it (Deployment, StatefulSet, Job) then creates a replacement, which the scheduler places on some other node — the tainted node has been filtered out for it. - If the taint is removed before expiry, the eviction is cancelled. A node that flaps Ready → NotReady → Ready inside the window costs you nothing. - `0` (or a negative value) means evict at once. - The field is **only** consulted for `NoExecute`. Setting it alongside `NoSchedule` is legal YAML but has no effect, which is a common source of confusion in review. ## Why every pod already has one The node lifecycle controller automatically taints nodes on condition changes. The two that matter most are `node.kubernetes.io/not-ready` (the node reports NotReady) and `node.kubernetes.io/unreachable` (the node controller has stopped hearing from the kubelet), both with effect `NoExecute`. If pods had no tolerations for these, a single missed heartbeat would immediately delete every pod on the node. To avoid that, the `DefaultTolerationSeconds` admission plugin in the API server injects tolerations for both keys with `tolerationSeconds: 300` into any pod that does not already declare them. That 300 seconds is where the familiar "pods take about five minutes to move after a node dies" behaviour comes from — and it is *in addition to* the node-monitor grace period the controller waits before deciding the node is unreachable at all, so real-world failover is often closer to six minutes. The default is cluster-wide and tunable through the API server flags `--default-not-ready-toleration-seconds` and `--default-unreachable-toleration-seconds`. A pod can override it individually by declaring its own toleration for those keys with a smaller value, which is the standard trick for latency-sensitive stateless services that want faster failover. Lowering it globally is risky: short values make the cluster reschedule aggressively during transient control-plane or network hiccups, causing eviction storms exactly when the cluster is least healthy. ## Eviction here bypasses PodDisruptionBudgets This catches people out. There are two very different things called "eviction": - **API-initiated eviction** — what `kubectl drain` and descheduling tools use. It goes through the Eviction subresource, respects PodDisruptionBudgets, and can be refused. - **Taint-manager eviction** — a plain pod deletion by the controller. It is not budget-aware. A node going NotReady is an involuntary disruption; the pods are already gone in practice, so blocking their removal would only leave the API object stranded. Do not rely on a PodDisruptionBudget to protect a workload from taint-based eviction. ## StatefulSets and the stuck-terminating case With an unreachable node the kubelet cannot confirm the pod is gone, so the pod object may sit in `Terminating` indefinitely. For a Deployment this is cosmetic, but a StatefulSet will not create the replacement `web-0` while the old `web-0` might still be running and holding a volume — the at-most-one guarantee demands it. Recovery is either deleting the Node object (which lets the API server force-delete its pods) or an explicit force delete. This is why NoExecute timing discussions almost always turn into a StatefulSet discussion in interviews. ## Practical uses Operators taint a node `maintenance=true:NoExecute` to push workloads off before a kernel patch; a hardware-health agent taints a node with failing disks; a cost controller taints spot/preemptible nodes on a termination notice with a short `tolerationSeconds` so pods have a bounded window to drain. In all of these, `tolerationSeconds` on the pods that *do* tolerate the taint is how you express "you may stay a bit longer, but not forever".