skip to content

A Terraform CI job exports TF_VAR_app_version before running terraform apply, but every apply keeps deploying the version pinned in a committed app.auto.tfvars. Why is the environment variable losing, and what do you change?

level: seniorimportance: should knowfreq 42%

answer

  1. environment is a fallback, not an override
  2. auto-loaded file outranks TF_VAR_
  3. no warning is printed either way
  4. fix at the file, not the tier
  5. injected names must never be committed

basics

~20 s

TF_VAR_ environment variables sit near the bottom of Terraform's precedence order, above only the variable block default, so any auto-loaded *.auto.tfvars file overrides them. Remove the pin from the committed file, or inject the value with -var instead.

solid answer

~40 s

Nothing is broken — `TF_VAR_` is the weakest real source Terraform has. It ranks above only the `default`, and `app.auto.tfvars` is auto-loaded at a higher tier, so the committed value silently wins and no warning is printed. There are three fixes, in order of preference: delete the pin from the committed file so the injected value is the only source; rename the file so it is no longer auto-loaded and pass it deliberately with `-var-file` when you do want it; or promote the injected value to the command line with `-var="app_version=$APP_VERSION"`, which outranks every file. I would also make it unrepeatable — a check that no `*.auto.tfvars` in the repository assigns any variable the pipeline injects, because the failure mode here is silence, not an error.

code

bash · 7 lines
bash
# fails the build if any committed variable file pins a CI-injected variable
INJECTED='app_version|db_password|image_tag'
if grep -REn "^[[:space:]]*($INJECTED)[[:space:]]*=" \
     --include='*.auto.tfvars' --include='terraform.tfvars' . ; then
  echo "error: injected variable assigned in a committed tfvars file" >&2
  exit 1
fi

go deeper

for a junior

Recall that TF_VAR_ ranks below tfvars files, so an auto-loaded file wins; do not assume the runner or the export is at fault.

for a middle

Explain the full order out loud, show which tier each of the two sources occupies, and describe the three fixes and their precedence consequences.

for a senior

Frame it as a silent-success incident: the pipeline was green while shipping the wrong version, so name the structural fix — injected variables never committed, plus a check that enforces it.

for a principal

Own the policy that prevents the class of bug: which values are repository-owned versus run-injected across every root module, and how an apply's inputs are recorded so it can be explained after the fact.

## What actually happened Terraform applies variable sources in a fixed order and lets later sources win: the `variable` block default first, then `TF_VAR_` environment variables, then `terraform.tfvars`, then `terraform.tfvars.json`, then `*.auto.tfvars` files in lexical order, then `-var`/`-var-file` on the command line. `app.auto.tfvars` is two tiers above the environment, so it wins. Terraform does not warn about this, because it is not an error condition — both sources supplied a valid value and the higher one was used. This catches people because every other tool they use treats the environment as an override. In Terraform it is explicitly documented as a *fallback* for the other ways of defining variables. ## Why it usually appears in CI and not locally The environment tier is the one place where a laptop and a runner routinely differ. A developer exports nothing and never notices the file is authoritative; the pipeline exports the value it thinks is authoritative and inherits the same file from the repository. The symptom is not a failed run — it is a green apply that shipped the wrong artifact version, which is far worse, because the pipeline reports success. ## The three fixes, and when each is right **Remove the pin.** If `app_version` is always supplied by the pipeline, it must not have a committed value at all. Delete it from `app.auto.tfvars`, and consider declaring the variable with no `default` so a run that forgets to supply it fails loudly instead of quietly deploying a stale version. **Rename so it is not auto-loaded.** `app.auto.tfvars` → `app.tfvars`, then pass `-var-file=app.tfvars` only where you actually want it. This is the general remedy for "this file applies in contexts I did not intend": auto-loading should carry only values that are true for every run from that directory. **Promote the injection to the command line.** `-var="app_version=$APP_VERSION"` beats every file. It is the quickest fix and the least self-documenting one — the value now exists only in the job's command line, so anyone reading the repository cannot tell what will be deployed. It is also a poor channel for secrets: command lines land in shell history, in the runner's process list, and often in the job log. ```bash # loses to any auto-loaded file export TF_VAR_app_version="$GIT_SHA" terraform apply -input=false -auto-approve # wins over every file, but is invisible to a repository reader terraform apply -input=false -auto-approve -var="app_version=$GIT_SHA" ``` ## Making it impossible to reintroduce Because the failure is silent, a convention plus a check beats remembering: - Decide which variables are *injected* (release identifiers, secrets, anything per-run) and which are *committed* (sizes, CIDRs, environment names). - Add a CI step that greps every `*.auto.tfvars` and `terraform.tfvars` in the repository for the injected names and fails the build if any of them appears. It is a few lines and it catches the exact regression that caused the incident. - Prefer directory-per-environment layouts, where the auto-loaded file next to the configuration cannot bleed into another environment's run. - If you keep the `TF_VAR_` channel, keep the injected variable defaultless so an absent value is an error rather than a fallback. ## Diagnosing the same class of bug next time There is no command that reports where a value came from, so walk the tiers in order, in the environment that actually ran Terraform: dump `env | grep TF_VAR_` inside the job, list the working directory for `terraform.tfvars*` and `*.auto.tfvars*` (remembering that `-chdir` changes which directory that is), and print the full Terraform command line including anything a wrapper script appends. One of those three will hold the winner. A temporary `output` echoing the value, or `terraform console`, confirms what the configuration resolved before you change anything. ## The wider lesson A variable value that can come from five places needs a *policy* about which place, not just a working pipeline. The strongest version of that policy is that the repository is the source of truth for everything except secrets and per-run identifiers, and that anything injected at run time is recorded in the job log by name so the apply can be explained afterwards.

  • Why not simply pass every value with -var and avoid the whole problem?
    Because the values then live only in a command line. Reviewers cannot see what production will get by reading the repository, long option lists get copied and mutated between jobs, and secrets passed this way appear in shell history, the process list and often the job log. `-var` is the right tool for a genuinely per-run value or a deliberate one-off override; it is a poor place for the standing configuration of an environment, which belongs in a reviewed file.
  • Would running terraform validate or a plan-time check have caught this before the apply?
    No. Both sources were valid, so nothing failed — the plan simply showed the committed version, which is exactly what a reviewer skimming a green plan is least likely to question. The detections that work are structural: keep the injected variable out of every committed tfvars file and enforce that with a repository check, or declare it without a default so an unsupplied value errors instead of falling back.
  • The same repository also injects TF_VAR_db_password. Does this bug put the password at risk?
    Not directly — a committed file overriding it would break the connection loudly rather than leak anything. The real risk is the fix: moving the secret to `-var` on the command line exposes it in the process list and job logs. Keep secrets on the `TF_VAR_` channel sourced from the pipeline's secret store, and make sure no committed file assigns that variable name at all.

saying these in an interview costs you the question

  • Assumes CI environment variables override committed tfvars files
  • Concludes the export or the runner is broken
  • Fixes it by deleting the environment variable entirely
  • Thinks Terraform warns when two sources set one variable
  • Moves a secret onto the command line to win precedence

context

resolving links…