When `kubectl drain` refuses to run, what do its `--ignore-daemonsets`, `--delete-emptydir-data` and `--force` flags each override, and what does each cost?

## Why `kubectl drain` refuses in the first place Before `kubectl drain` evicts anything, it lists every pod on the node and runs each through a set of **filters**. Three filters can produce a *fatal* result, and one fatal pod stops the whole drain: nothing is evicted, although the node has already been cordoned. The error message names the flag that would override it: - `cannot delete DaemonSet-managed Pods (use --ignore-daemonsets to ignore)` - `cannot delete Pods with local storage (use --delete-emptydir-data to override)` - `cannot delete Pods that declare no controller (use --force to override)` Pods that have already finished (phase `Succeeded` or `Failed`) never trip these filters, and **mirror pods** (static pods) are skipped silently. The refusal is deliberate: each category represents something that eviction cannot handle safely on its own, so the tool makes you say out loud that you accept the consequence. | Flag | What it overrides | What actually happens to the pod | The cost you accept | |---|---|---|---| | `--ignore-daemonsets` | the DaemonSet check | **nothing**: the pod is left running | the node keeps its per-node agents until it shuts down | | `--delete-emptydir-data` | the `emptyDir` check | evicted; the `emptyDir` contents are deleted | any scratch or cache data in that volume is lost | | `--force` | the "no controller" check | evicted like any other pod; **nobody recreates it** | the workload is gone until someone recreates it by hand | ## `--ignore-daemonsets`: skip, do not evict A DaemonSet runs one pod per eligible node, and its pods tolerate the `node.kubernetes.io/unschedulable` taint that cordoning adds. If drain evicted a DaemonSet pod, the DaemonSet controller would simply create it again on the same node. So drain never evicts running DaemonSet pods at all. The flag only means "I know these are here; proceed with the rest". This is almost always what you want: the log shipper, the CNI agent and the node exporter should keep running until the machine goes down. One edge case: a pod whose owner reference points at a DaemonSet that no longer exists is an orphan, and drain removes it only with `--force`. ## `--delete-emptydir-data`: the data dies with the pod An `emptyDir` volume lives for as long as the pod lives on that node. Evicting the pod deletes the directory, so drain stops unless you acknowledge the loss. In practice most `emptyDir` volumes are caches, sockets or temp files, and the flag is routine. It is not routine when an application keeps something it cannot rebuild there, for example a write-ahead buffer that has not yet been shipped. Note what the filter does **not** look at: it checks for `emptyDir` only. A `hostPath` volume does not block the drain, and neither does a PersistentVolumeClaim. A pod bound to a **local PersistentVolume** (a `local` volume with node affinity) is evicted normally, but its replacement can only schedule on the same node, so it stays `Pending` until that node is uncordoned. ## `--force`: bare pods are not rescheduled A pod created directly, with no ReplicaSet, StatefulSet, Job or other controller in `metadata.ownerReferences`, has nobody to recreate it. Drain refuses to remove it because removal is permanent. `--force` removes it anyway. Before passing it, find out who created the pod and why: a debugging pod can go, while a hand-launched migration run should be finished or rerun under a Job first. ## Flags that are *not* about safety checks Three more flags matter when a drain misbehaves: 1. `--grace-period` overrides each pod's `terminationGracePeriodSeconds`; the default of `-1` means "use the pod's own value". 2. `--timeout` bounds the whole drain; the default `0` means wait forever. 3. `--disable-eviction` makes drain use a plain `DELETE` instead of the Eviction API, which **skips PodDisruptionBudget checks** entirely. It is an emergency tool, not a way to make routine drains faster. A preview avoids surprises. `--dry-run=client` does not cordon the node; it lists the pods drain would evict and still reports the pods that would block it: ```bash kubectl drain node-b-41 --ignore-daemonsets --delete-emptydir-data --dry-run=client ``` ## What interviewers listen for - That `--ignore-daemonsets` **leaves** the pods, rather than deleting them. - That `--force` is about pods with no controller, not about disruption budgets or stuck pods. - That each flag is a decision about data or availability, taken per node, not a default to paste into every script.