skip to content

With Terratest's `test_structure` package, how do you re-run only the validation step of a slow test without redeploying the module each time?

level: middleimportance: nice to knowfreq 25%

answer

  1. split the test into named parts
  2. an environment variable turns one off
  3. skipped code cannot hand over variables
  4. data travels through files on disk
  5. the temp copy stops happening

basics

~10 s

Wrap deploy, validate and teardown in test_structure.RunTestStage, then set SKIP_<stage> for each stage to skip. Save options to disk in the deploy stage and load them later, because a skipped stage's function never runs.

solid answer

~40 s

I split the test into named stages with `test_structure.RunTestStage(t, "validate", func() { ... })`, with the teardown stage deferred. A stage is skipped when an environment variable named `SKIP_<stage>` is non-empty, so `SKIP_teardown=true go test` leaves the module deployed, and `SKIP_deploy=true SKIP_teardown=true go test` then runs just the validation. Skipped code cannot pass variables along, so the deploy stage calls `SaveTerraformOptions` and later stages call `LoadTerraformOptions`, which read a JSON file under `.test-data`. When any `SKIP_` variable is set, `CopyTerraformFolderToTemp` also stops copying and returns the original folder, so state survives between runs. In CI I set none of these, and every stage runs against a temp copy.

code

go · 24 lines
go
func TestVpcModule(t *testing.T) {
	ctx := t.Context()
	dir := test_structure.CopyTerraformFolderToTemp(t, "..", "examples/vpc")

	defer test_structure.RunTestStage(t, "teardown", func() {
		options := test_structure.LoadTerraformOptions(t, dir)
		terraform.DestroyContext(t, ctx, options)
	})

	test_structure.RunTestStage(t, "deploy", func() {
		options := terraform.WithDefaultRetryableErrors(t, &terraform.Options{
			TerraformDir: dir,
			Vars:         map[string]any{"name": "vpc-test-" + strings.ToLower(random.UniqueID())},
		})
		test_structure.SaveTerraformOptions(t, dir, options)
		terraform.InitAndApplyContext(t, ctx, options)
	})

	test_structure.RunTestStage(t, "validate", func() {
		options := test_structure.LoadTerraformOptions(t, dir)
		url := terraform.OutputContext(t, ctx, options, "endpoint_url")
		http_helper.HTTPGetWithRetryContext(t, ctx, url, &tls.Config{}, 200, "ok", 30, 5*time.Second)
	})
}

go deeper

for a junior

Recall that test_structure splits a Terratest test into named stages and that a SKIP_ environment variable turns one off, so local iteration does not redeploy every time.

for a middle

Explain why stages save options to disk, where the unique ID must be generated, and which command sequence deploys, iterates and finally tears down.

for a senior

Call out the side effect: any SKIP_ prefixed variable disables the temp copy, which is unsafe for parallel runs. Also mention secrets, since saved options are logged and written to a plain file.

for a principal

Consider whether stages justify their extra structure for a given suite, and set a rule that CI never sets SKIP_ variables, since one misplaced variable leaks resources or breaks isolation.

A full Terratest run is slow, because deploying and destroying real resources is slow. While you are working on the validation code, that cost repeats for every small edit. Terratest's `test_structure` package (import path `modules/test-structure`) addresses this by splitting one test into named stages that you can switch off individually. ## How a stage works `test_structure.RunTestStage(t, "validate", func() { ... })` runs the function unless an environment variable named `SKIP_validate` is set. - **The name is yours.** `deploy`, `validate` and `teardown` are conventions, not reserved words. - **Any non-empty value skips.** The check only compares against the empty string, so `SKIP_validate=false` also skips the stage. - **Teardown stays deferred.** `defer test_structure.RunTestStage(t, "teardown", ...)` is registered first, so it runs on failure, just as a plain deferred destroy would. - **With no variables, every stage runs.** CI gets the complete test with no code change. ## Passing data between runs Inside one process, closures can share variables. Stages cannot rely on that, because a skipped stage's function never runs and each `go test` is a new process. So the data goes to disk: - `SaveTerraformOptions` and `LoadTerraformOptions` handle the `terraform.Options` value. - `SaveString`, `LoadString`, `SaveInt` and `LoadInt` handle single values. - `SaveArtifactID` and `LoadArtifactID` replace the older `SaveAmiId` and `LoadAmiId` names. - Everything lands under `<folder>/.test-data/`, for example `TerraformOptions.json`, and `CleanupTestDataFolder` removes it. This decides where the unique ID belongs. Generate it **inside** the deploy stage and save it with the options. Generated at the top of the test, it would change on every run and no longer match what is deployed. ## A local session A typical working session has three kinds of command: 1. Deploy and validate, and leave the result running: `SKIP_teardown=true go test -count=1 -timeout 30m -run TestVpcModule`. 2. Edit the checks and repeat only the validation: `SKIP_deploy=true SKIP_teardown=true go test -count=1 -timeout 30m -run TestVpcModule`. 3. Finish by running only the teardown: `SKIP_deploy=true SKIP_validate=true go test -count=1 -timeout 30m -run TestVpcModule`. The third command is the one people forget, and until it runs the resources keep costing money. The `-count=1` matters here too: Terratest's docs point out that Go can serve a cached result when the Go code has not changed, which is often the case when you are editing Terraform files. ## When stages earn their keep Stages add structure, so they pay off when deployment is the slow part. Terratest's own end-to-end example has several: `build_ami`, `deploy_terraform`, `validate`, then deferred `logs`, `cleanup_terraform` and `cleanup_ami`. Reverse-order deferral means the logs are fetched before Terraform destroys the instance. A small module that applies in a minute gains little from this, and a plain deferred destroy stays easier to read. The stage names also become part of how your team works, so keep them consistent across a suite. ## What else changes when a SKIP variable is set `test_structure.SkipStageEnvVarSet()` returns true if **any** environment variable starts with `SKIP_`, whatever the rest of its name. `CopyTerraformFolderToTemp` calls it and, when it is true, skips the copy and returns the original module folder. That is deliberate, because state and `.test-data` have to survive between runs. The source comment states the assumption: this is local development with no other tests running concurrently. | | No `SKIP_` variable | Any `SKIP_` variable | |---|---|---| | Stages that run | All | All except those named | | `CopyTerraformFolderToTemp` returns | A fresh temp copy | The original module folder | | Safe for parallel tests | Yes | No | ## Traps - **An unrelated variable in CI.** A job that happens to define something like `SKIP_LINT` switches off the temp copy, so parallel tests share one folder and overwrite each other's files. - **`SKIP_teardown` set in CI for debugging.** Every run then leaves its resources behind. - **Secrets in saved options.** `SaveTerraformOptions` logs the options as JSON and writes them to a plain file, `Vars` included. Keep secrets out of `Vars`, and keep `.test-data` out of version control. - **Overwriting saved options.** `SaveTerraformOptions` replaces an existing file. Its doc comment warns this can duplicate resources, because a re-run of deploy with a new ID creates a second set. `SaveTerraformOptionsIfNotPresent` keeps the first value.

  • Why does Terratest stop copying the module to a temp folder once a `SKIP_` variable is set?
    A fresh temp copy on each run would lose the state and `.test-data` the earlier run left behind, so later stages would find nothing. `CopyTerraformFolderToTemp` therefore returns the original folder, assuming local development with no concurrent tests. That assumption is why a stray `SKIP_` variable in CI is dangerous for parallel suites.

  • Stages are skipped with a flag passed to go test on the command line.
  • A later stage can read local variables set inside a skipped earlier stage.
  • Setting SKIP_teardown in CI is a harmless way to make debugging easier.
  • Only a variable named after a declared stage changes Terratest's behaviour.
  • SKIP_validate=false makes Terratest run the validate stage.