How would you run a scheduled Terraform drift-detection job in CI, and how should the job decide whether to raise an alert?

## Why the job exists A plan run on a pull request tells you what your change will do. It says nothing about the weeks in between, when someone clicks in the console during an incident, an operator widens a security group "temporarily", or a service mutates a field on your behalf. Terraform will notice all of it — but only the next time somebody happens to plan that root module, which for a stable module might be months. A scheduled drift job turns "we find out eventually" into "we find out tomorrow morning". ## The mechanics The job is an ordinary Terraform run with the interactivity stripped out: ```bash set -euo pipefail terraform init -input=false -no-color set +e terraform plan -detailed-exitcode -input=false -no-color -lock-timeout=2m > plan.txt 2>&1 code=$? set -e case "$code" in 0) echo "no drift" ;; 2) echo "DRIFT"; cat plan.txt; exit 1 ;; *) echo "terraform failed"; cat plan.txt; exit "$code" ;; esac ``` `-detailed-exitcode` is the whole trick: without it, a successful plan exits `0` whether or not it found anything, and the job has to scrape human-readable output. With it, plan returns `0` for no changes, `1` for an error, `2` for changes present — the only three values, and each means something different to your alerting. ## Deciding what to alert on **Exit 2 is not automatically drift.** A normal plan diffs configuration against refreshed state, so exit 2 means "the world does not match this code", which can be either real drift *or* a commit that was merged and never applied. Two ways to keep the signal honest: - Run against the revision that was last successfully applied, not the tip of the branch, so unapplied code cannot appear in the diff. - Or run `terraform plan -refresh-only`, whose diff is state-versus-reality only and therefore isolates out-of-band change specifically. Most teams run the normal plan, because "deployed infrastructure does not match main" is worth knowing regardless of which of the two causes it is — but the alert text should say both are possible so the responder does not go hunting for a console culprit that never existed. **Exit 1 is an operations signal, not a drift signal.** Expired credentials, a throttled provider, a removed upstream module, a lock held by someone else. Treat it separately: a single failure is noise, the same module failing every night is a broken pipeline nobody is watching. ## Operating it without making enemies - **State locks.** A drift plan takes the same lock as a real apply. Pass `-lock-timeout` so a plan that collides with a deploy waits instead of failing instantly, and classify a lock timeout as "retry later", never as drift. - **Refresh cost.** Every run re-reads every resource in state. Multiply that by every root module in the estate on the same cron minute and you will find your provider's rate limiter. Stagger the schedule. - **Credentials.** The job only needs to read, so give it a read-scoped role — ideally through short-lived federated credentials rather than a static key sitting in the scheduler. - **Never auto-apply.** "Detected drift, so I reverted it" is how a scheduled job destroys the emergency capacity someone added at 2am. Detection and remediation are separate decisions with a human in between. - **Redaction.** Plan output can contain values you would rather not paste into a chat channel. Post a link to the job, or a summarised resource list, rather than the raw diff. ## Making the output usable A wall of plan text at 6am gets muted within a fortnight. What works: name the root module, list the addresses that changed, and route to the team that owns that module. `terraform show -json` over a saved plan gives you machine-readable output if you want to build a proper summary rather than shipping raw text. ## What to say out loud Schedule a non-interactive plan per root module, branch on `-detailed-exitcode`, alert a human on `2` with the resource addresses, keep `1` in a separate operational bucket, hold a `-lock-timeout` so you never fight a real deploy, give the job read-scoped credentials, and never let it apply anything by itself.