Most teams refuse to run a full apply-and-destroy integration test on every commit to their infrastructure repository. Why, and how would you decide what runs on a pull request versus nightly versus before a release?

## The four forces Ask why not run everything every time and the answer is not one reason but four, and a strong answer names them separately because each has a different mitigation. **Wall-clock time.** Real infrastructure is slow: managed databases, clusters, certificates and DNS propagation are measured in tens of minutes, and teardown is often as slow as creation. A pull-request gate that takes forty minutes is not a gate; it is a queue, and people learn to merge around it. **Money.** Every run bills for what it creates, and the bill scales with how realistic the environment is. Worse, failed teardown bills *forever* until somebody notices. A dozen commits a day multiplied by a realistic environment is a line item somebody will eventually ask about. **Contention and quota.** A test account has finite limits: addresses, instances, network constructs, API rate. Ten concurrent pull requests each building an environment will exhaust something, and then the failures are about each other rather than about any change. Globally unique names collide. You can buy your way partly out with per-run accounts from a pool, but that is real platform investment, not a pipeline setting. **Non-determinism.** Capacity errors, throttling and eventual consistency make the run fail sometimes for reasons unrelated to the code. This is the force that matters most, because a blocking gate that is wrong one time in five teaches the team to re-run until green — at which point it blocks nothing while still costing everything. ## The placement principle Use one rule: **what blocks a merge must be fast and deterministic; what is slow or intermittent runs on a schedule with a named owner.** Everything else follows. **On the pull request** — target single-digit minutes, no credentials beyond read-only: - static checks: parse, schema, lint, secret and misconfiguration scanning - a generated plan, published where the reviewer can read it, with assertions on the diff: destructive actions, blast-radius size, mandatory tags, forbidden resource classes - module tests against faked providers for anything with real logic - contract checks on shared module interfaces **Nightly or on a schedule** — pays the money once per day: - build and destroy a real environment in a throwaway account - a small behavioural check that the thing actually works, not just that it was created - a janitor sweep for orphaned resources, and a drift check against real environments **Before a release** — the highest-fidelity, lowest-frequency run: - the full path at realistic scale and configuration - an upgrade rehearsal: deploy the version production currently runs, then apply the new one, so you test the *transition* and not just the greenfield build. This is the level most teams skip and the one that finds forced-replacement surprises. - a destroy rehearsal, because the delete path is code too ## Escape hatches worth naming - **Change-scoped triggers.** If only one component changed, deploy only that component's environment. Most commits touch a small area, and this turns an unaffordable run into an affordable one. - **A pool of accounts.** Hand each run its own account or project from a pool so contention and quota stop being shared problems, and teardown failure is bounded by recycling the account. - **Label-driven opt-in.** Let an author request the expensive run on a risky pull request without imposing it on every typo fix. - **Retry budgets by class.** Automatically retry the known-transient provider errors a bounded number of times; never retry a genuine assertion failure. ## Reporting, and the honest part Say out loud what you are accepting. Moving the real apply off the merge path means a change can merge and later fail at the top level. That is a deliberate trade: fast feedback and a trustworthy gate in exchange for a delayed, owned discovery of provider-level problems. Make it survivable — the nightly result alerts a person, the failure is triaged into regression versus provider noise, and a broken nightly blocks the next release rather than being carried indefinitely. ``` PR (minutes) static + plan assertions + module tests blocking nightly (an hour) real apply + behaviour + janitor owned, alerts release (hours) full scale + upgrade + destroy rehearsal blocks the release ``` ## What the interviewer is listening for Not the schedule — the reasoning. That flakiness, not cost, is the strongest argument against gating on the expensive level. That the upgrade path deserves its own rehearsal because greenfield builds hide replacement risk. And that any level you cannot afford to keep green should be reduced in scope rather than left permanently amber.