skip to content

helm upgrade keeps fighting an operator over a field both set — how do you fix it?

level: seniorimportance: should knowfreq 46%

answer

  1. two writers, one field
  2. which kind of ownership is this
  3. the symptom differs by apply mode
  4. the chart should stop declaring it

basics

~20 s

Stop declaring the field in the chart — whichever side is not its owner must leave it out. Otherwise Helm 4's server-side apply fails the upgrade with a field-manager conflict, and Helm 3's client-side merge silently overwrites the controller on every run.

solid answer

~50 s

The fix is editorial, not a flag: the template must stop emitting the field, usually guarded by a value so the chart writes it only when nothing else owns it. Understand the two symptoms first. Under Helm 4, server-side apply is the default write path, so an upgrade that sets a field another manager owns is rejected with a conflict and the release stays put. Under the Helm 3 client-side three-way merge — which a release first installed by Helm 3 keeps, because `--server-side` on upgrade defaults to `auto` and inherits the previous method — there is no conflict: Helm patches the field back to the chart's value, the controller writes its own again, and the object flip-flops. `--force-conflicts` exists to take the field over deliberately; reach for it only when the chart genuinely is the owner, and note it is unrelated to `--force`, a deprecated alias of `--force-replace`.

code

bash · 3 lines
bash
helm upgrade ledger ./ledger-umbrella            # fails: conflict on spec.instances
helm get manifest ledger | grep -n 'instances'   # confirm the chart still declares it
helm upgrade ledger ./ledger-umbrella --force-conflicts   # deliberate takeover only

go deeper

for a junior

Recognise the pattern: if a chart and something else in the cluster both set the same field, the chart should stop setting it, usually behind a values toggle in the template.

for a middle

Explain the two write paths — server-side apply reporting a field-manager conflict, and the older client-side merge patching the value back — and which flag belongs to which mechanism.

for a senior

Diagnose it end to end from managedFields and the release's stored manifest, know why auto makes two clusters behave differently, and treat --force-conflicts as a one-off act rather than a pipeline flag.

for a principal

Set the rule that any field a controller computes is not the chart's to declare, and decide how charts across the estate are migrated onto a single apply mode so the failure mode is consistent.

### First, decide which "ownership" you are talking about Three different mechanisms wear this word, and mixing them up is the most common way this conversation goes wrong: * **Release ownership** — the `app.kubernetes.io/managed-by: Helm` label and the `meta.helm.sh/release-name` and `meta.helm.sh/release-namespace` annotations Helm puts on the objects a release owns. This is about *which release* an object belongs to. `helm upgrade --take-ownership` (available since Helm 3.17) is the flag for that problem: adopting existing objects into a release. * **Field ownership** — server-side apply's field managers, recorded in the object's `metadata.managedFields`. This is about *which writer* owns each individual field. `--force-conflicts` is the flag for that problem. * **Owner references** — the parent/child links Kubernetes uses for garbage collection, which is how a controller's own created objects are cleaned up. A fight between a chart and a controller over one field is the second kind, and only the second kind. Using `--take-ownership` on it does nothing; using `--force` does something else entirely. ### Why Helm 4 changed the symptom Helm 4 made server-side apply the default write path. On `install`, `--server-side` is a boolean defaulting to true. On `upgrade` and `rollback` it is a *string* defaulting to `auto`, which inherits the apply method of the previous release. That inheritance produces a genuinely confusing field report: a release first installed by Helm 3 and later upgraded with a Helm 4 client stays on the old client-side path, so two clusters running the same chart and the same CLI can behave differently. Check the release's history before you believe either symptom. **Under server-side apply**, each writer records the fields it sets. When the chart declares `spec.instances` on the ledger's custom resource and the operator has already claimed that field, the apply comes back as a conflict and the upgrade fails outright. That is unpleasant but honest — the cluster tells you two writers disagree. **Under the Helm 3 client-side three-way merge**, Helm computes a patch from the previous revision's stored manifest, the live object and the newly rendered manifest, and patches the field back to the chart's value. No error. The controller reconciles and writes its own value again. The object oscillates, `helm get manifest` disagrees with the live object between runs, and every upgrade produces a spurious diff. ### The fix: one field, one owner Whoever is not the owner must not emit the field at all. Not "emit the same value" — not emit it. A chart that writes the value the controller currently happens to want is still claiming the field, and will still fight the moment the controller changes its mind. In a chart this is a conditional in the template, gated by a value: ```yaml spec: {{- if not .Values.autoscaling.enabled }} replicas: {{ .Values.replicaCount }} {{- end }} ``` The same pattern applies to any field a controller computes: a scaled replica count, an image tag rewritten after a managed version migration, a resource request set by a sizing controller, a status-derived annotation. Under server-side apply there is a second half to the fix: when Helm stops declaring a field it previously owned, it releases it, so the object converges on the controller's value rather than keeping a stale one. ### Diagnosing it Read the object's `metadata.managedFields` — it names every manager and the exact fields each one owns, which turns "something keeps changing this" into a definite answer. Compare it against `helm get manifest <release>`, which shows what the release believes it applied. If the field appears in the chart's manifest *and* under another manager in `managedFields`, you have found the fight. For the flip-flopping client-side case, the giveaway is that the value is correct immediately after an upgrade and wrong minutes later, on a period that matches the controller's resync rather than anything Helm does. ### When `--force-conflicts` is right When the chart genuinely is the owner and something else claimed the field by accident — a one-off `kubectl` edit during an incident, a migration from a tool that has since been removed. `--force-conflicts` makes Helm take the field over and become its manager. It is a deliberate, one-time act, not a way to make the upgrade command stop complaining; wiring it into a pipeline permanently guarantees you will overwrite a controller that had a good reason for its value. It is also mutually exclusive with `--force-replace`, and note the trap: `--force` is a deprecated alias of `--force-replace`, which is about recreating resources, and has nothing to do with server-side apply conflicts. A candidate who says "just use `--force`" has named the wrong flag for the wrong mechanism. ### The judgment behind it The real answer to "who wins" is that nobody should have to win. Two writers on one field is a design defect, and the chart is usually the side that should yield, because the controller is reacting to information — load, health, migration progress — that the chart's author did not have at render time.

  • How is `--force-conflicts` different from `--take-ownership`?
    They address two different ownerships. `--take-ownership` adopts objects that are not part of this release — it rewrites the `meta.helm.sh/release-name` and `release-namespace` annotations so the release claims them. `--force-conflicts` overrides server-side apply *field* managers on objects the release already owns. An object can be perfectly owned by your release and still have individual fields owned by a controller.
  • The same chart conflicts on one cluster and silently overwrites on another. Why?
    Because on upgrade `--server-side` defaults to `auto`, which inherits the previous release's apply method. A release originally installed by Helm 3, or by Helm 4 with server-side apply turned off, stays on the client-side three-way merge and keeps overwriting; a release first installed by Helm 4 uses server-side apply and reports the conflict. Check the release history, not just the CLI version.
  • After you remove a field from the chart, does the old value linger on the live object?
    Under server-side apply, no — dropping a field from the applied configuration releases Helm's claim on it, and the field converges on whatever the remaining owner sets or is removed if nobody owns it. Verify with the object's managedFields after the first upgrade, because a client-side release inherited through `auto` behaves differently and may need a deliberate move to server-side apply.

saying these in an interview costs you the question

  • Says --force resolves server-side apply conflicts
  • Confuses release ownership annotations with field managers
  • Wires --force-conflicts permanently into the pipeline
  • Sets the same value in the chart instead of omitting the field
  • Thinks conflicts and flip-flopping are the same symptom
  • Assumes the Helm CLI version alone decides the apply mode

context