Compare kubectl create, kubectl replace and kubectl apply for updating a live Kubernetes object, and explain what server-side apply changed about tracking field ownership and conflicts.
answer
- create = fail if exists; replace = full overwrite; apply = merge
- last-applied-configuration annotation = client-side three-way merge
- managedFields = per-field owner, server-side
- 409 Conflict names the contested field; --force-conflicts takes it
- omit a field to relinquish ownership
basics
~20 screate fails if the object exists; replace overwrites the whole object, dropping fields you omitted; apply merges your manifest with the live object. Server-side apply moves that merge into the API server and records per-field owners in metadata.managedFields, so a conflicting write is rejected until you force it.
solid answer
~60 s**create** makes an object and errors if it already exists. **replace** does a full overwrite: whatever your file omits is removed, and it needs the current resourceVersion, so it clobbers concurrent changes or fails. **apply** is declarative: it merges your manifest into the live object and leaves fields owned by others alone. Classic client-side apply did this in kubectl using a three-way merge between your file, the live object, and the last-applied-configuration annotation — the annotation existed so kubectl could tell "you removed this field" from "you never mentioned it". It was fragile: the annotation could be stale or missing, and merging happened on whichever client version you ran. **Server-side apply** moves the merge into the API server. Each applier sends its manifest with a field manager name; the server records which manager owns which field in metadata.managedFields. If you try to set a field another manager owns to a different value, you get a 409 Conflict listing the field, and you must either drop it, take ownership with --force-conflicts, or fix the real owner. Removing a field from your manifest relinquishes it — no annotation needed.
code
bash · 6 lineskubectl apply --server-side --field-manager=ci-pipeline -f deploy.yaml
kubectl get deploy web -o yaml --show-managed-fields | grep -A20 managedFields
# conflict path
kubectl apply --server-side --field-manager=ci-pipeline -f deploy.yaml
# Apply failed with 1 conflict: conflict with "hpa-controller": .spec.replicas
kubectl apply --server-side --force-conflicts -f deploy.yamlgo deeper
Distinguish the three verbs and state that apply is the declarative, repeatable one used in pipelines.
Explain three-way merge and the last-applied-configuration annotation, then how managedFields and 409 conflicts replace it.
Focus on multi-writer ownership: which manager should own which field, why forcing conflicts causes flapping, and how to name field managers in automation.
Treat field ownership as an interface contract between platform components, and define org-wide rules on who owns replicas, resources and annotations before conflicts appear in production.
## Three verbs, three semantics **create** is imperative: POST the object, fail with AlreadyExists if it is there. Fine for one-shot setup, useless in a pipeline that runs repeatedly. **replace** is a full PUT. The object becomes exactly what you sent, so any field you omitted disappears — including things another component legitimately set. It also requires the current resourceVersion, so it either fails or overwrites whatever changed between your read and your write. Use it when you truly want "the object is now exactly this". **apply** is declarative and idempotent: create if absent, otherwise merge. It is the verb GitOps and CI use because running it twice is a no-op and it does not disturb fields it does not mention. ## Why apply needed a memory Merging alone cannot express deletion. If your manifest no longer mentions a field, did you delete it, or did you never own it? Client-side apply solved this with the `kubectl.kubernetes.io/last-applied-configuration` annotation: kubectl stored the manifest it applied last time and did a three-way merge across previous manifest, current live object, and new manifest. Fields present before and absent now were removed; fields never in your manifest were left alone. The weaknesses were real. The annotation could be absent (object created with create) or stale (someone used replace), merge logic lived in the client so different kubectl versions could disagree, and only kubectl participated — controllers and operators writing to the same object were invisible to the scheme. The annotation also duplicated the whole object, sometimes blowing size limits. ## Server-side apply Server-side apply makes apply a first-class API operation (`PATCH` with content type `application/apply-patch+yaml`). The client sends the manifest it cares about plus a **field manager** identity; the API server does the merge and records ownership in `metadata.managedFields` — a list of entries, one per manager and operation, each holding the set of field paths that manager owns. Consequences: - **Conflicts are explicit.** If manager `argocd` owns `spec.replicas: 3` and you apply `spec.replicas: 5` as manager `kubectl`, you get 409 Conflict naming the field. You can drop the field, pass `--force-conflicts` to take ownership, or change the real owner. Previously that write would have silently won and been silently reverted later. - **Removal is by omission.** Drop a field from your manifest and you release it; if nobody else owns it, it is removed from the object. - **Multiple writers coexist.** An autoscaler can own `spec.replicas` while your manifest owns the pod template, without either fighting the other, provided your manifest omits replicas. - **Lists need merge keys.** Structured merge relies on schema-declared list types (map/set/atomic) so per-item ownership works; atomic fields are owned wholesale. ## Practical guidance Use `kubectl apply --server-side` (and set `--field-manager` to a meaningful name in automation) so ownership is attributable. Do not reflexively pass `--force-conflicts` in CI — a conflict is information: another actor believes it owns that field. Investigate before overriding. If you migrate a long-lived object from client-side to server-side apply, the API server reconciles the old annotation into managedFields on first use, and the annotation stops being authoritative.
- You run apply with a manifest that hardcodes spec.replicas while a HorizontalPodAutoscaler also manages it. What happens, and what is the right fix?With server-side apply you get a 409 Conflict naming spec.replicas, because the autoscaler's field manager owns it. Forcing the conflict makes your value win momentarily and the autoscaler will move it again, producing flapping. The correct fix is to remove replicas from the manifest entirely so ownership stays with the autoscaler.
- Why can kubectl apply not simply diff your manifest against the live object?Because the live object contains fields set by defaulting and by other controllers that your manifest never mentioned. A two-way diff would read every such field as one you want removed. That is why deletion detection needs either the last-applied-configuration annotation (client-side) or recorded per-field ownership in managedFields (server-side).
saying these in an interview costs you the question
- Saying apply and replace are interchangeable
- Claiming apply deletes any field missing from the manifest, regardless of ownership
- Not knowing what last-applied-configuration was for, or thinking it is still authoritative under server-side apply
- Treating --force-conflicts as the standard way to make CI green
- Believing managedFields is user-authored configuration rather than server-maintained bookkeeping