skip to content

How do you find and inspect a specific subset of objects across a large Kubernetes namespace — for example every pod not in the Running phase on one particular node — using kubectl selectors and output formats?

level: middleimportance: should knowfreq 42%

answer

  1. -l labels: =, !=, in, notin, existence
  2. --field-selector: whitelist only, = and !=
  3. status.phase, spec.nodeName for pods
  4. selectors server-side; jsonpath client-side
  5. -L / --show-labels to spot label mistakes

basics

~20 s

Use -l for label selectors (equality and set-based) and --field-selector for a small whitelist of built-in fields such as status.phase and spec.nodeName; both filter server-side. Shape output with -o wide, custom-columns, jsonpath, and --sort-by, which are client-side.

solid answer

~40 s

Two filters exist and they are not the same thing. **Label selectors** (`-l`) match `metadata.labels`, are arbitrary and user-defined, and support equality (`app=api`), inequality (`tier!=cache`), set membership (`-l 'env in (stage,prod)'`) and existence (`-l 'canary'`, `-l '!canary'`). Services and controllers use the same grammar, so `-l` is how you ask which pods a Service actually selects. **Field selectors** (`--field-selector`) match fields of the object itself, but only a **whitelisted set** per resource — for pods, `metadata.name`, `metadata.namespace`, `spec.nodeName`, `status.phase`, `spec.restartPolicy`, `spec.schedulerName`, `status.podIP`; for events, `involvedObject.*`, `reason`, `type`. Anything else errors. So: `kubectl get pods -A --field-selector 'status.phase!=Running,spec.nodeName=node-3'`. Output shaping is separate and client-side: `-o wide`, `-o custom-columns=...`, `-o jsonpath=...`, `--sort-by=.status.startTime`. jsonpath filters after everything is fetched, so on huge clusters push filtering server-side first.

code

bash · 5 lines
bash
kubectl get pods -A --field-selector 'status.phase!=Running'
kubectl get pods -A --field-selector spec.nodeName=node-3 -o wide
kubectl -n prod get pods -l 'app=api,env in (prod)' --show-labels
kubectl -n prod get pods --sort-by=.status.containerStatuses[0].restartCount
kubectl -n prod get pods -o custom-columns='NAME:.metadata.name,NODE:.spec.nodeName,IP:.status.podIP'

go deeper

for a junior

Know -l for labels and -o wide, and that -n and -A control scope.

for a middle

Distinguish label from field selectors, recall the supported pod fields, and shape output with custom-columns or jsonpath.

for a senior

Reason about server-side versus client-side filtering cost on large clusters, and use selectors to compare what a Service selects with what exists.

for a principal

Guide conventions — a consistent label taxonomy is what makes selector-based triage and blast-radius queries possible at all.

## Why two selector mechanisms Labels are metadata you attach; fields are structure the API already has. Label selectors are the Kubernetes-native grouping mechanism — Services, Deployments, NetworkPolicies and PodDisruptionBudgets all select by label, so debugging with the same grammar tells you exactly what those controllers see. Field selectors exist because some queries ("pods on this node", "events about this object") cannot be expressed with labels and would otherwise force clients to download everything. ## Label selector grammar - Equality: `-l app=api`, `-l app=api,tier=backend` (comma is AND). - Inequality: `-l 'tier!=cache'`. - Set-based: `-l 'env in (stage,prod)'`, `-l 'release notin (canary)'`. - Existence: `-l 'team'` (has the key), `-l '!team'` (lacks it). Useful moves during triage: `kubectl get pods -l app=api -o wide` to see all replicas of one workload at once; `kubectl logs -l app=api --prefix --tail=20` to sample logs across them; and comparing a Service's selector with the pods that selector actually returns — a mismatch after a label change is a classic cause of a Service with no backends. ## Field selector limits The crucial fact is that **only specific fields are supported**, and they differ per resource; a selector on an arbitrary nested field such as a container image is not supported and returns an error. Supported and worth memorising: - Any resource: `metadata.name`, `metadata.namespace`. - Pods: `status.phase`, `spec.nodeName`, `status.podIP`, `spec.restartPolicy`, `spec.schedulerName`. - Events: `involvedObject.name`, `involvedObject.kind`, `involvedObject.namespace`, `reason`, `type`, `source`. - Nodes: `metadata.name`, `spec.unschedulable`. Operators are limited to `=`, `==` and `!=`, combined with commas as AND. Two high-value triage queries: `kubectl get pods -A --field-selector status.phase=Pending` to find everything that will not schedule cluster-wide, and `kubectl get pods -A --field-selector spec.nodeName=node-3` to see everything a suspect node is carrying before you drain it. ## Output formats - `-o wide` adds node, pod IP, nominated node and readiness gates. - `-o custom-columns='NAME:.metadata.name,NODE:.spec.nodeName,RESTARTS:.status.containerStatuses[0].restartCount'` builds exactly the table you want. - `-o jsonpath=...` for scripting; `-o json` piped into jq is often more readable if jq is available. - `--sort-by=.status.containerStatuses[0].restartCount` ranks the worst offenders; `--sort-by=.metadata.creationTimestamp` orders by age. - `--show-labels` and `-L team,env` display label values as columns — handy when you suspect a labelling mistake. All of these run **after** the objects arrive on your machine. On a cluster with tens of thousands of pods, fetching every pod as JSON and filtering locally transfers everything and can strain the API server; push the filter server-side with `-l` and `--field-selector` first, then shape locally. ## Scope flags `-A` / `--all-namespaces` widens the search, `-n` narrows it. `kubectl get all` is misleadingly named — it covers only a common subset of resource types and omits ConfigMaps, Secrets, Ingresses and custom resources, so never treat it as an inventory. `kubectl api-resources` lists what exists and whether it is namespaced, which is the honest way to enumerate.

  • Why does a field selector on a pod's container image fail while one on spec.nodeName works?
    Field selectors are not a general query language; the API server supports only a whitelisted set of fields per resource type, because each supported field must be indexed or specially handled. spec.nodeName is on that list for pods, arbitrary nested container fields are not, so the request is rejected. To filter by image you must fetch the objects and filter client-side with jsonpath or jq.

saying these in an interview costs you the question

  • Believing field selectors work on any field of the object
  • Confusing label selectors with field selectors
  • Fetching every pod as JSON and filtering locally on a huge cluster instead of filtering server-side
  • Treating kubectl get all as a complete inventory of a namespace
  • Not checking labels when a Service unexpectedly has no endpoints

context

resolving links…