skip to content

When one pod of a Kubernetes Deployment running a nightly ledger-reconciliation batch misbehaves, how would you use kubectl label to isolate it for debugging, and what happens afterwards?

level: seniorimportance: nice to knowfreq 30%

answer

  1. change, don't just add
  2. release removes the ownerReference
  3. replacement restores replica count
  4. unmanaged pods block drain
  5. re-adoption triggers a scale-down

basics

~20 s

Overwrite a label the ReplicaSet selects on. The ReplicaSet releases the pod and creates a replacement, and a Service that selects on that label stops routing to it. The pod keeps running unmanaged, so you must delete it yourself.

solid answer

~40 s

First read the ReplicaSet's selector, then change one label it requires, ideally one the Service also selects on. For example: `kubectl label pod ledger-worker-7f9c4 app.kubernetes.io/instance=ledger-quarantine --overwrite`. The ReplicaSet controller sees an owned pod that no longer matches, **releases** it by removing its controller `ownerReference`, and creates a replacement to get back to `replicas`. The pod keeps running for debugging. It is now **unmanaged**: nothing updates or deletes it, it still uses node resources, and `kubectl drain` refuses it without `--force`. Add a `quarantine=true` marker so these pods can be listed, and delete the pod when you are done. Restoring the original label makes the ReplicaSet re-adopt it and then delete a surplus pod, which may not be the one you debugged.

code

bash · 5 lines
bash
kubectl get rs -n ledger -l app.kubernetes.io/name=ledger-reconciler -o wide
kubectl label pod ledger-worker-7f9c4 -n ledger app.kubernetes.io/instance=ledger-quarantine quarantine=true --overwrite
kubectl annotate pod ledger-worker-7f9c4 -n ledger example.com/quarantined-by=oncall-ledger
kubectl get pods -A -l quarantine=true
kubectl delete pod ledger-worker-7f9c4 -n ledger

go deeper

for a junior

Recall that changing a pod's labels can take it out of its ReplicaSet, and that the ReplicaSet then starts a new pod.

for a middle

Explain release and adoption: which label must change, why adding a label is not enough, and what a Service selecting on that label does.

for a senior

Show the operational follow-through: mark and track quarantined pods, know that drain stops on unmanaged pods, and avoid the re-adoption scale-down.

for a principal

Decide whether quarantine by relabelling is an allowed runbook step in a shared cluster, and what marker, owner and expiry rules it needs.

## The situation A Deployment runs the workers of a **nightly ledger-reconciliation batch** in a 140-node cluster shared by 22 product teams. One worker pod behaves strangely: slow queries and odd log lines. Deleting it would destroy the evidence. You want to **keep it running for debugging** while the Deployment restores full capacity. Labels give you a way to do that. ## How relabelling isolates a pod A ReplicaSet decides which pods it owns by its **label selector**. Its controller constantly reconciles: - An owned pod that **still matches** the selector is counted. - An owned pod that **no longer matches** is **released**: the controller patches the pod to remove its controller `ownerReference`. - A pod with **no controller** whose labels match is **adopted**. So the steps are: 1. Read the selector: `kubectl get rs -n ledger -o wide` (or `kubectl describe deployment`) shows it. It includes your identity labels plus the controller-added `pod-template-hash`. 2. Overwrite **one label the selector uses**, and preferably one the Service selects on too: `kubectl label pod ledger-worker-7f9c4 -n ledger app.kubernetes.io/instance=ledger-quarantine --overwrite` 3. Add a marker that says who did it and why, for example a `quarantine=true` label (this key is not in the selector) plus a note in an annotation. ## What happens next | Actor | Reaction | |---|---| | ReplicaSet controller | Releases the pod, sees fewer pods than `replicas`, and creates a replacement | | Service with a selector on the changed label | Stops choosing the pod as a backend, so it gets no new traffic | | The pod itself | Keeps running. The kubelet still applies its `restartPolicy` to the containers | | Deployment | Nothing changes. It still reports full availability through the new pod | A few details matter: - **Adding** a label rarely isolates a pod. Most selectors only require labels to exist or to equal a value, and extra labels still match. You must **change or remove** a label the selector requires. - Do not relabel it to a value that **another** controller's selector matches. Any controller with a matching selector can adopt an orphan. - The quarantined pod still uses its node's CPU and memory and still counts against any pod quota set on the namespace. - The replacement pod is scheduled like any new pod, so the batch keeps its full worker count while you investigate. ## The part people forget: the orphan A released pod is an **unmanaged pod**. Nothing deletes it, reschedules it or updates it. Weeks later this matters: - `kubectl drain` refuses unmanaged pods. It reports *Pods that declare no controller (use --force to override)* and stops before evicting anything. In this cluster, the platform team's routine 13-minute node drain aborted at the start because of one forgotten quarantine pod. - It keeps running the old image through every later rollout. - Anything that still selects it by a label you did **not** change keeps counting it. Hygiene that avoids this: - Always add the `quarantine=true` marker, so `kubectl get pods -A -l quarantine=true` lists every isolated pod. - Record the owner and a deadline in an annotation, and review the list regularly. - Delete the pod as soon as the debugging session ends. ## Putting it back If you **restore** the original label, the pod matches again and has no controller, so the ReplicaSet **re-adopts** it. The ReplicaSet now has one pod more than `replicas` and **deletes one pod** to converge, and it chooses which one by its own ranking. That may be a healthy pod rather than the one you debugged. It is usually cleaner to delete the quarantined pod than to restore its label. ## When this is not the right tool - For a crash you must reproduce, `kubectl debug` with a copy of the pod may be simpler, because the original pod stays under its ReplicaSet. - If the selector has only one label besides `pod-template-hash`, changing that label is enough. If a Service selects on a different label, change that one too, or the pod keeps receiving traffic while you debug it. - For batch pods owned by a **Job**, the Job's generated selector uses a controller-uid label rather than your app labels, so changing an app label does not release the pod. Choose the technique per controller.

  • Why might adding a quarantine=true label to a Kubernetes pod leave it in its ReplicaSet?
    A ReplicaSet selector lists labels a pod must have, or must have with a given value. It rarely forbids extra labels. Adding a new key leaves every requirement satisfied, so the pod still matches and stays owned. You must change or remove a label the selector requires, unless the selector has a `DoesNotExist` requirement on the key you add.
  • What happens if you relabel a quarantined Kubernetes pod so it matches its ReplicaSet's selector again?
    The pod has no controller and its labels match, so the ReplicaSet adopts it. The ReplicaSet now has one pod above `replicas` and deletes one pod to converge. Its own ranking picks the victim, which may be a healthy pod. Deleting the quarantined pod is usually the cleaner ending.

saying these in an interview costs you the question

  • Expecting the ReplicaSet to delete a pod that stops matching its selector
  • Assuming adding any new label removes the pod from its ReplicaSet
  • Forgetting that a released pod keeps using node resources indefinitely
  • Believing a released pod can never be adopted again
  • Relabelling to a value another controller's selector matches