skip to content

How does client-side `kubectl apply` combine the last-applied configuration, the live Kubernetes object and your file to decide what to change?

level: middleimportance: must knowfreq 58%

answer

  1. three documents, not two
  2. removed versus never declared
  3. memory lives in an annotation
  4. declared fields always win
  5. replicas reset after autoscaling

basics

~20 s

Client-side kubectl apply runs a three-way merge. Fields in your file are set on the live object, fields in the last-applied record but gone from the file are deleted, and fields in neither are left alone.

solid answer

~50 s

Client-side `kubectl apply` compares three documents. They are the **new file**, the **live object**, and the **last-applied configuration** that kubectl stored in the `kubectl.kubernetes.io/last-applied-configuration` annotation on the previous apply. Any field in the new file whose value differs from the live one is set. Any field that was in last-applied but is missing from the new file is deleted, because that is how apply knows you *removed* it. Fields in neither document, typically written by controllers or `kubectl patch`, are untouched. kubectl sends the result as a **strategic merge patch** for built-in kinds, where lists like `containers` merge by `name`. For kinds without that metadata, such as custom resources, it sends a JSON merge patch. It then rewrites the annotation to the new file. The classic trap is `replicas` in a file for an autoscaled Deployment: every apply resets it.

code

yaml · 17 lines
yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: records-portal
spec:
  replicas: 7
  selector:
    matchLabels:
      app: records-portal
  template:
    metadata:
      labels:
        app: records-portal
    spec:
      containers:
        - name: portal
          image: registry.example/records-portal:4.18.2

go deeper

for a junior

Remember that apply keeps a copy of what you last applied on the object, and uses it to notice fields you deleted from the file.

for a middle

Walk through the three rules (set declared fields, delete dropped ones, ignore undeclared ones) and say which patch format kubectl sends for built-in kinds and for custom resources.

for a senior

Explain the replicas reset under an autoscaler, why deleting the line drops the Deployment to one replica, and how edit-last-applied avoids it. Mention the 256 KiB annotation ceiling.

for a principal

Argue when the annotation-based model stops scaling for a platform, because of mixed tools, huge objects and shared fields, and what moving the fleet to server-side apply involves.

## The three inputs A declarative tool has to answer one hard question: when a field is missing from my file, did I delete it, or did I never care about it? Client-side `kubectl apply`, the default mode, answers it by keeping a memory. Each apply compares three documents: - **Modified**: the YAML file you are applying now. - **Current**: the live object read from the API server, including everything controllers and people have written since. - **Original**: the configuration from your *previous* apply. kubectl stores it as JSON in the annotation `kubectl.kubernetes.io/last-applied-configuration` on the object itself. That is why it is called a **three-way merge**. A two-way diff of file against live object cannot tell "removed by me" from "added by someone else". ## The rules 1. **Declared and different**: if a field in the file differs from the live value, the patch sets it to the file's value. This holds even if a controller changed it on purpose. 2. **Previously declared, now absent**: if a field is in the original but not in the file, the patch deletes it from the live object. 3. **Never declared**: if a field is in neither the file nor the original, it is not in the patch at all, so whatever other actors wrote survives. 4. **Record the new baseline**: the annotation is updated to the file you just applied, so it becomes the original next time. ## A worked example The clinical-records portal's Deployment `records-portal` runs on a 38-node cluster with a spot and an on-demand pool. | Field | Last-applied | Live | New file | Result | |---|---|---|---|---| | `spec.replicas` | 7 | 13 (scaled up during a 310 ms p99 spike) | 7 | set back to **7** | | `nodeSelector` `node-pool: on-demand` | present | present | removed | **deleted** | | toleration added with `kubectl patch` | absent | present | absent | **kept** | | container image | `4.18.1` | `4.18.1` | `4.18.2` | set to **4.18.2** | The first row is the famous trap. An autoscaler raised `replicas` to 13 for good reason, and the next routine deploy quietly cut it back to 7 because the file still declares the field. ## How the patch is encoded kubectl does not send the merged object. It sends a **patch**: - For built-in kinds whose types carry merge metadata, it sends a **strategic merge patch**. Lists such as `spec.template.spec.containers` are merged by a key (`name`), so changing one container's image leaves a sidecar alone. - Some built-in lists are marked atomic, for example Pod `tolerations`. If your file declares them, the whole list is replaced. - For kinds without that metadata, notably custom resources, kubectl sends a **JSON merge patch**, in which any list you declare replaces the live list wholesale. ## Where it breaks - **No annotation.** An object made with `kubectl create` (without `--save-config`) or by another tool has no original. The first apply cannot detect removals, and kubectl warns and adds the annotation. - **Removing `replicas` to fix the autoscaler fight.** Removing it from the file makes it "previously declared, now absent", so apply *deletes* it. The server then defaults `replicas` to 1, and the portal drops to one Pod until the autoscaler reacts. The safe route is to first remove `replicas` from the stored record with `kubectl apply edit-last-applied` (or `set-last-applied`), then from the file. - **Annotation size.** The annotation holds a full copy of the file, and all annotations on an object together are limited to 256 KiB. A very large ConfigMap cannot be client-side applied. - **Mixed tools.** Anything that changes the annotation, or applies with a different tool, corrupts the "original", and deletions become unpredictable. Server-side apply removes all four problems by tracking ownership per field on the server instead of in one annotation. ```bash kubectl apply view-last-applied deployment/records-portal kubectl apply edit-last-applied deployment/records-portal ```

  • How do you safely stop declaring `replicas` in a client-side-applied Deployment that an autoscaler now controls?
    Do not just delete the line and apply. The field is in last-applied, so apply would delete it and the server would default `replicas` to 1. First drop it from the stored record with `kubectl apply edit-last-applied` (or `set-last-applied` with a file that lacks it), then remove it from the manifest. Later applies will treat `replicas` as never declared and leave it to the autoscaler.
  • Why does changing one container's image in a file not remove a sidecar that another tool injected?
    For a Deployment, kubectl sends a strategic merge patch. The `containers` list is merged by its `name` key, so the patch only touches the entry whose name matches your file. A sidecar that is in neither your file nor last-applied is not mentioned in the patch and survives. With a JSON merge patch, the whole list would be replaced.

saying these in an interview costs you the question

  • apply compares only the file with the live object
  • fields removed from the file are simply ignored
  • apply never overwrites values that controllers changed
  • the last-applied record is stored somewhere in etcd outside the object
  • removing replicas from the file is always a no-op