skip to content

What goes wrong when a deploy job runs helm upgrade --install without --version?

level: seniorimportance: should knowfreq 56%

answer

  1. The reference is a query, not an identity
  2. A range still floats
  3. Same commit, two different charts
  4. The packaging job supplies the pin
  5. A local path pins nothing

basics

~20 s

A chart reference with no --version resolves to whatever the repository currently offers as newest, so the same pipeline run twice can deploy two different charts. The deploy job has no fixed input and the result is not reproducible.

solid answer

~40 s

`charts-repo/payments-ledger` with no `--version` means *whatever that repository says is newest right now*. Somebody publishing a chart between two runs changes what your unchanged pipeline deploys, environments silently diverge, and re-running an old pipeline to reproduce last week's state gives you this week's chart. `--version` accepts a constraint, not only an exact version, so `^2.0.0` still floats - pin the exact string, `2.14.3`. That string should come from the packaging job that produced `payments-ledger-2.14.3.tgz` and be passed to the deploy job as an explicit input, with the job failing when it is empty. Verify after the fact with `helm list`, whose CHART column shows the name and version the release was installed from. Note what pinning does not cover: it fixes templates and defaults, not the images they name.

code

bash · 5 lines
bash
CHART_VERSION="${CHART_VERSION:?deploy requires an exact chart version}"

helm upgrade --install ledger charts-repo/payments-ledger \
  --version "$CHART_VERSION" \
  -n payments -f envs/prod.yaml

go deeper

for a junior

Learn the shape of the flag: a chart reference without --version means newest available, and an exact version makes the command deploy the same chart every time. Being able to read the CHART column of helm list is the bar here.

for a middle

Explain that --version takes a constraint, so a range floats too, and that the exact string should be produced upstream by packaging rather than chosen by the deploy job.

for a senior

Diagnose the consequences in a live estate: silent environment drift, an incident whose cause was somebody else's publish, and a rollback that no longer resolves to the chart it used to. Say how you verify what actually shipped.

for a principal

Own the reproducibility contract across the fleet: where chart versions originate, how they are recorded with each deployment, and how you stop pipelines from acquiring an implicit dependency on whatever a repository served that afternoon.

A chart reference like `charts-repo/payments-ledger` is not an artifact identity, it is a query. Without `--version`, Helm resolves it against the repository's index - or the registry's tags for an `oci://` reference - and takes the newest stable version available at the moment the command runs. The deploy job therefore has no fixed input. The same commit, the same pipeline definition and the same values can install chart 2.14.3 in the morning and 2.15.0 in the afternoon because somebody merged and published a chart change in between. ## What --version actually accepts `--version` takes a constraint, not only an exact version. `2.14.3` pins one chart. `^2.0.0` or `~2.14` are ranges, and a range floats exactly the way no flag at all floats, just within narrower walls; a chart published an hour ago that satisfies the range will be selected. `--devel` widens selection further by admitting pre-release versions, which is a debugging convenience and not something a production deploy job should carry. If you want reproducibility, the value must be a single exact version. ## What floating costs you Four things go wrong, and a senior answer names them concretely. First, deploys stop being reproducible. Re-running last month's pipeline to restore last month's state gives you today's chart with last month's values, a combination nobody ever tested. Second, environments drift without anyone changing anything. Staging deployed on Tuesday got 2.14.3; production deployed on Thursday got 2.15.0; the pipeline definition is identical in both, so the difference is invisible in the repository and only visible in the cluster. Third, the blast radius of a chart change decouples from the act of merging it. A chart author publishes 2.15.0 and the next unrelated deploy of any consumer - triggered by a config change, a retry, a scheduled run - picks it up. The person who caused the change and the person who suffers it are different people on different days, which is the worst shape an incident can have. Fourth, rollback becomes ambiguous. Rolling the application back means redeploying the pipeline that used to be green, but that pipeline no longer resolves to the chart it once did. ## Where the pinned version comes from The pin is not invented by the deploy job; it is produced upstream. A packaging job runs `helm package` on a tag and emits `payments-ledger-2.14.3.tgz`, and that version string is the artifact identity. The deploy job's contract is then simple: it takes a chart name and an exact version as an input, and fails loudly when the input is empty rather than defaulting to latest. Anything that consumes charts downstream - including a controller that is handed a pinned chart version rather than a rendered manifest - takes the same string, which is what makes one version walkable across environments at all. ## The local-path trap Many pipelines never touch a repository: the deploy job checks out the source and runs `helm upgrade --install ledger ./charts/payments-ledger`. This looks pinned because it is local, and it is the opposite. `--version` selects among the versions a repository offers, so it pins nothing against a directory; the deployed chart is whatever the checkout happened to contain, and a job that checks out `main` deploys chart content that was never packaged, never linted as a tarball and cannot be named after the fact. If the deploy job builds the chart it deploys, the pipeline has no boundary between build and deploy. ## Verifying what actually shipped `helm list -n payments` shows a CHART column - `payments-ledger-2.14.3` - and an APP VERSION column, which are different facts: the first is the chart's own version, the second is the `appVersion` string the chart carries about the application. `helm history` shows the chart per revision, so you can see the exact revision at which the chart version moved. Reconciling that against what the pipeline believes it deployed is the first check in any *why is prod different* investigation. ## A worked incident The payments-ledger deploy takes about seven minutes. A production deploy triggered for an unrelated values change resolved `charts-repo/payments-ledger` to 2.15.0, published forty minutes earlier, which renamed the key a mounted Secret is read from. The upgrade applied cleanly, the seven minutes elapsed, and the ledger pods failed on startup - with a pipeline diff that showed only a two-line values change. The fix was one flag and one guard: pin `--version` to the exact string the packaging job produced, and fail the job when that parameter is unset. ## What pinning does not do Pinning the chart version fixes the templates, the defaults and the packaged dependencies. It does not fix the images those templates name. If the chart's default image tag is mutable, two deploys of chart 2.14.3 render identical YAML and can still run different code, because the tag resolved to a different digest. Chart pinning and image pinning are separate disciplines and you need both.

  • Does passing --version '^2.0.0' make the deploy reproducible?
    No. `--version` accepts a constraint, and a range is resolved against the repository at run time just like no flag at all, only within narrower bounds. Any 2.x published between two runs can be selected. A range is a policy about what you are willing to accept, not a record of what you deployed; reproducibility needs a single exact version, produced by the packaging job and handed to the deploy job as an input.
  • The deploy job installs from a checked-out chart directory. Where does the pin go?
    There is none to add - `--version` selects among versions a repository offers and does nothing against a path. The deployed content is whatever the checkout contained. Either package the chart in an upstream job and deploy the packaged version by name, or accept that the pipeline has no build/deploy boundary and that you cannot say afterwards which chart ran. The first option is what makes rollback and audit answerable.
  • You pinned the chart version and two deploys still behaved differently. What else moved?
    Most likely the images. A pinned chart fixes templates and defaults, not the code they run: a mutable image tag renders identically and resolves to a different digest on each pull. Values supplied outside the chart can also differ between runs. Check the rendered image references and the values the release recorded before assuming the chart is at fault.

saying these in an interview costs you the question

  • Believing a chart reference alone identifies one chart
  • Treating a semver range as a pin
  • Assuming a local chart path is reproducible
  • Reading the APP VERSION column as the chart version
  • Letting the deploy job default to latest when unset
  • Thinking a pinned chart also pins the container images

context

resolving links…