Why is a Kubernetes Deployment's spec.selector immutable in apps/v1, and how do you change which pods a Deployment selects?

middleshould knowfreq 56%

answer

selector is the ownership boundary
release and adopt by labels
field is immutable on update
template labels must satisfy selector
new Deployment, then delete old

basics

~20 s

A Deployment's spec.selector decides which ReplicaSets and pods it owns. Changing it in place would orphan them or adopt other objects, so apps/v1 rejects the change as immutable. To change it, create a new Deployment and delete the old one.

solid answer

~40 s

In apps/v1, a Deployment's spec.selector is required and non-empty. The template labels must satisfy it, and after creation any change is rejected with spec.selector: ... field is immutable. The reason is that the selector is the ownership boundary: controllers release objects that stop matching and adopt matching orphans, so editing it in place could release every running ReplicaSet and pod, or adopt objects that belong to something else. The Deployment controller also adds pod-template-hash to each ReplicaSet's selector so ReplicaSets never overlap. To change the selector, create a Deployment with a new name and the new labels, move the Service to it, then delete the old one, or delete and recreate and accept a gap. Avoid the problem by selecting only on stable identity labels such as app.kubernetes.io/name and app.kubernetes.io/instance, never on version.

In `apps/v1`, a Deployment's `spec.selector` is required and non-empty. The template labels must satisfy it, and after creation any change is rejected with `spec.selector: ... field is immutable`. The reason is that the selector is the **ownership boundary**: controllers release objects that stop matching and adopt matching orphans, so editing it in place could release every running ReplicaSet and pod, or adopt objects that belong to something else. The Deployment controller also adds `pod-template-hash` to each ReplicaSet's selector so ReplicaSets never overlap. To change the selector, create a Deployment with a new name and the new labels, move the Service to it, then delete the old one, or delete and recreate and accept a gap. Avoid the problem by selecting only on stable identity labels such as `app.kubernetes.io/name` and `app.kubernetes.io/instance`, never on version.

code

bash · 6 lines

bash

kubectl get deployment ledger-reconciler -n ledger -o jsonpath='{.spec.selector}{"\n"}'
# editing matchLabels in the manifest and re-applying is rejected:
# spec.selector ... field is immutable
kubectl apply -f ledger-reconciler-v2.yaml
kubectl rollout status deployment/ledger-reconciler-v2 -n ledger
kubectl delete deployment ledger-reconciler -n ledger

go deeper

for a junior

Remember that a Deployment's selector cannot be edited after creation and must match the pod template labels.

for a middle

Explain adoption and release by selector, why that makes an in-place selector change dangerous, and what pod-template-hash is for.

for a senior

Plan the side-by-side replacement with a Service cutover, and catch tools that inject labels into selectors before they break every apply.

for a principal

Set a selector labelling standard, with stable identity keys only, so no team ever needs to replace a workload to fix one.

What the selector of a Deployment means

A Deployment manages pods indirectly. It owns ReplicaSets, and each ReplicaSet owns pods. The link at every step is a label selector:

spec.selector on the Deployment says which ReplicaSets and pods belong to it.
spec.template.metadata.labels are the labels every new pod gets.
For each ReplicaSet it creates, the Deployment controller adds a pod-template-hash label to the ReplicaSet's selector, to its pod template, and to the ReplicaSet itself. The hash is computed from the pod template, so the old and new ReplicaSets of one Deployment never select each other's pods during a rollout.

In apps/v1 the selector is required, and validation enforces three rules:

The selector must not be empty. The error is empty selector is invalid for deployment, because an empty selector would match every pod in the namespace.
The template's labels must satisfy the selector. Otherwise the error is `selector` does not match template `labels`, because the Deployment would create pods it could not count.
After creation, spec.selector cannot change. An update that changes it is rejected with spec.selector: Invalid value: ...: field is immutable. The same rule applies to ReplicaSets, StatefulSets and DaemonSets in apps/v1.

Why immutability is the right call

The selector is the Deployment's ownership boundary. Controllers use it in two directions:

Adopt: a pod or ReplicaSet with no controlling owner, whose labels match, can be claimed.
Release: an owned object whose labels no longer match is let go (its controller ownerReference is removed).

If the selector could change in place, one edit would do one of two things. It could release every existing ReplicaSet and pod at once, leaving orphans running while new ones are created, so you briefly run double capacity. Or it could start adopting orphaned pods and ReplicaSets that were never meant to be part of this Deployment. Old API versions allowed selector changes, and these surprises are why apps/v1 forbids them.

You change...	Allowed?	Effect
A template label that the selector does not use	Yes	New pod template, so a rollout starts
A template label that the selector uses	Rejected	The template would no longer match the selector
`spec.selector` (add, remove or change)	Rejected	`field is immutable`
Labels on the Deployment object itself	Yes	No effect on which pods are selected

How to change the selector anyway

You cannot edit the selector, so you replace the object:

Side-by-side (no downtime). Create a Deployment with a new name and the new selector and template labels. Wait for it to become ready, move traffic by pointing the Service's selector at a label both sets of pods share (or at the new label), then delete the old Deployment.
Delete and recreate (with a gap). Delete the Deployment and apply it again under the same name. The pods go away with it by default, so accept the capacity gap or plan the deletion carefully.

For the nightly ledger-reconciliation batch workers, the side-by-side route suits a daytime change, when the batch is not running and a short double capacity is cheap.

Choosing selector labels so you never need to

Select on identity: app.kubernetes.io/name plus app.kubernetes.io/instance is a good pair.
Never select on anything that changes per release, such as app.kubernetes.io/version, an image tag or a git sha. The first upgrade would need a selector change.
Put volatile information in extra template labels. The template may carry more labels than the selector requires.
Watch tools that inject labels. A manifest transformer that adds a "common label" to selectors as well as to metadata will change spec.selector on the next apply, and the API server rejects it for every existing Deployment.

yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ledger-reconciler
  namespace: ledger
spec:
  replicas: 3
  selector:
    matchLabels:
      app.kubernetes.io/name: ledger-reconciler
      app.kubernetes.io/instance: ledger-nightly
  template:
    metadata:
      labels:
        app.kubernetes.io/name: ledger-reconciler
        app.kubernetes.io/instance: ledger-nightly
        app.kubernetes.io/version: "4.12.1"
    spec:
      containers:
        - name: reconciler
          image: registry.example.com/ledger/reconciler:4.12.1

Here the version label lives only in the template. Upgrading to 4.12.2 is an ordinary rollout, and the selector never changes.

Checking before you apply

Run kubectl diff -f on the manifest. A changed selector block in the diff is a warning sign.
Compare kubectl get deployment -o jsonpath='{.spec.selector}' with what a tool renders, especially after upgrading that tool.

## What the selector of a Deployment means A Deployment manages pods indirectly. It owns **ReplicaSets**, and each ReplicaSet owns **pods**. The link at every step is a **label selector**: - `spec.selector` on the Deployment says which ReplicaSets and pods belong to it. - `spec.template.metadata.labels` are the labels every new pod gets. - For each ReplicaSet it creates, the **Deployment controller** adds a `pod-template-hash` label to the ReplicaSet's selector, to its pod template, and to the ReplicaSet itself. The hash is computed from the pod template, so the old and new ReplicaSets of one Deployment never select each other's pods during a rollout. In `apps/v1` the selector is **required**, and validation enforces three rules: 1. The selector must not be empty. The error is `empty selector is invalid for deployment`, because an empty selector would match every pod in the namespace. 2. The template's labels must satisfy the selector. Otherwise the error is `` `selector` does not match template `labels` ``, because the Deployment would create pods it could not count. 3. After creation, `spec.selector` **cannot change**. An update that changes it is rejected with `spec.selector: Invalid value: ...: field is immutable`. The same rule applies to ReplicaSets, StatefulSets and DaemonSets in `apps/v1`. ## Why immutability is the right call The selector is the Deployment's **ownership boundary**. Controllers use it in two directions: - **Adopt**: a pod or ReplicaSet with no controlling owner, whose labels match, can be claimed. - **Release**: an owned object whose labels no longer match is let go (its controller `ownerReference` is removed). If the selector could change in place, one edit would do one of two things. It could release every existing ReplicaSet and pod at once, leaving orphans running while new ones are created, so you briefly run double capacity. Or it could start adopting orphaned pods and ReplicaSets that were never meant to be part of this Deployment. Old API versions allowed selector changes, and these surprises are why `apps/v1` forbids them. | You change... | Allowed? | Effect | |---|---|---| | A template label that the selector does not use | Yes | New pod template, so a rollout starts | | A template label that the selector uses | Rejected | The template would no longer match the selector | | `spec.selector` (add, remove or change) | Rejected | `field is immutable` | | Labels on the Deployment object itself | Yes | No effect on which pods are selected | ## How to change the selector anyway You cannot edit the selector, so you **replace the object**: 1. **Side-by-side (no downtime).** Create a Deployment with a **new name** and the new selector and template labels. Wait for it to become ready, move traffic by pointing the Service's selector at a label both sets of pods share (or at the new label), then delete the old Deployment. 2. **Delete and recreate (with a gap).** Delete the Deployment and apply it again under the same name. The pods go away with it by default, so accept the capacity gap or plan the deletion carefully. For the nightly ledger-reconciliation batch workers, the side-by-side route suits a daytime change, when the batch is not running and a short double capacity is cheap. ## Choosing selector labels so you never need to - Select on **identity**: `app.kubernetes.io/name` plus `app.kubernetes.io/instance` is a good pair. - **Never** select on anything that changes per release, such as `app.kubernetes.io/version`, an image tag or a git sha. The first upgrade would need a selector change. - Put volatile information in extra template labels. The template may carry **more** labels than the selector requires. - Watch tools that inject labels. A manifest transformer that adds a "common label" to selectors as well as to metadata will change `spec.selector` on the next apply, and the API server rejects it for every existing Deployment. ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: ledger-reconciler namespace: ledger spec: replicas: 3 selector: matchLabels: app.kubernetes.io/name: ledger-reconciler app.kubernetes.io/instance: ledger-nightly template: metadata: labels: app.kubernetes.io/name: ledger-reconciler app.kubernetes.io/instance: ledger-nightly app.kubernetes.io/version: "4.12.1" spec: containers: - name: reconciler image: registry.example.com/ledger/reconciler:4.12.1 ``` Here the version label lives only in the template. Upgrading to `4.12.2` is an ordinary rollout, and the selector never changes. ## Checking before you apply - Run `kubectl diff -f` on the manifest. A changed `selector` block in the diff is a warning sign. - Compare `kubectl get deployment -o jsonpath='{.spec.selector}'` with what a tool renders, especially after upgrading that tool.

Why does a Kubernetes Deployment add pod-template-hash to each ReplicaSet it creates?
During a rollout, the old and new ReplicaSets have the same identity labels. The Deployment controller adds a `pod-template-hash` label, computed from the pod template, to each ReplicaSet's selector, its template and the ReplicaSet itself. Each ReplicaSet then selects only its own pods, so the two never count or adopt each other's pods.
Can you add a new label to a Kubernetes Deployment's pod template without touching spec.selector?
Yes. The template may carry more labels than the selector requires, as long as it still satisfies every selector requirement. Adding or changing a label the selector does not use changes the template, so the Deployment starts an ordinary rollout. Changing a label the selector does use is rejected, because the template would no longer match.

saying these in an interview costs you the question

Expecting kubectl apply to relabel running pods when the selector changes
Putting app.kubernetes.io/version or an image tag into the selector
Thinking template labels must equal the selector exactly, with no extras
Believing you can patch spec.selector if you use kubectl edit
Assuming the controller copies selector labels into the template for you

Why is a Kubernetes Deployment's spec.selector immutable in apps/v1, and how do you change which pods a Deployment selects?

§ answer

basics

solid answer

code

§ go deeper

▾level notes4 levels

▾deep dive~4 min

What the selector of a Deployment means

Why immutability is the right call

How to change the selector anyway

Choosing selector labels so you never need to

Checking before you apply

▾common follow-ups2 questions

▾red flags5 flags

What the selector of a Deployment means

Why immutability is the right call

How to change the selector anyway

Choosing selector labels so you never need to

Checking before you apply

answer

go deeper