skip to content

A Terratest test of a VPC module fails its endpoint check right after apply but passes on rerun: where do retries belong, and what should not be retried?

level: seniorimportance: should knowfreq 33%

answer

  1. two layers, two kinds of failure
  2. apply finished does not mean ready
  3. only listed patterns repeat
  4. one error type ends the loop
  5. the budget includes the destroy

basics

~10 s

Retry at two layers: terraform.WithDefaultRetryableErrors re-runs a Terraform command on listed transient errors, and http_helper or retry.DoWithRetryContext polls the live check. Deterministic failures should stop at once, and retries must fit the test timeout.

solid answer

~40 s

Two layers, because they fail differently. A successful `apply` says the API calls finished, not that the endpoint answers yet, so I poll with `http_helper.HTTPGetWithRetryContext` or `retry.DoWithRetryContext`, which repeats on any error except `retry.FatalError`. For Terraform commands, `terraform.WithDefaultRetryableErrors` sets `MaxRetries` to 3 and `TimeBetweenRetries` to 5 seconds, and repeats only failures matching its regex list, such as `Provider produced inconsistent result after apply`; anything else fails immediately. A bare `terraform.Options` retries nothing. What should not repeat is a deterministic failure: a validation error, or an endpoint answering with the wrong content. Broad patterns and huge counts turn those into slow failures and can hide a flaky defect. Everything must also fit inside `go test -timeout`, with room left for the destroy.

code

go · 15 lines
go
url := terraform.OutputContext(t, ctx, options, "endpoint_url")

retry.DoWithRetryContext(t, ctx, "GET "+url, 30, 5*time.Second, func() (string, error) {
	status, body, err := http_helper.HTTPGetContextE(t, ctx, url, &tls.Config{})
	if err != nil {
		return "", err // not answering yet: retry
	}
	if status != http.StatusOK {
		return "", fmt.Errorf("status %d", status) // not ready yet: retry
	}
	if body != expected {
		return "", retry.FatalError{Underlying: fmt.Errorf("wrong body: %s", body)} // will not heal
	}
	return body, nil
})

go deeper

for a junior

Recall that a finished apply does not mean a service is ready, and that Terratest has retry helpers for HTTP checks and for Terraform commands.

for a middle

Explain the two defaults: Terraform commands repeat only on listed patterns, while retry.DoWithRetryContext repeats on anything except retry.FatalError. Know that a bare Options value does not retry.

for a senior

Work out the retry budget against go test -timeout, including the destroy, and show how an over-generous retry can cause a leak. Say what you would mark as fatal and why.

for a principal

Set a team standard: shared retry settings, a review bar for new patterns, and retry counts watched as a signal, so that accumulating retries does not quietly hide unreliable modules.

Many cloud APIs are **eventually consistent**: a change they accept can take time to become visible everywhere. Live-account tests meet this often, and Terratest offers separate tools for the two places it shows up. This answer follows Terratest 1.0.1. ## Layer one: the Terraform command The command itself sometimes fails for reasons that heal on their own: a plugin download hits a network error, or a provider reads back a value that has not propagated yet. `terraform.Options` controls this with three fields, and `terraform.WithDefaultRetryableErrors` fills them in. | Field | Bare `terraform.Options` | After `WithDefaultRetryableErrors` | |---|---|---| | `RetryableTerraformErrors` | Empty | Regex list of known transient errors | | `MaxRetries` | 0 | 3 | | `TimeBetweenRetries` | 0 | 5 seconds | The behaviour has these details: - **Matching decides.** Each failure's output and error text are checked against the regexes. A match waits and repeats; no match is wrapped in a `retry.FatalError` and returned at once. - **What is listed.** The defaults include registry and plugin errors, connection resets and `Provider produced inconsistent result after apply`, which a source comment describes as usually an eventual consistency issue. - **It covers every Terraform command**, because all of them go through `RunTerraformCommandContextE`, so `init` and `destroy` benefit as well as `apply`. - **It returns a copy.** Use the returned value, because your original options are unchanged. - **It overwrites two fields.** `MaxRetries` and `TimeBetweenRetries` are set unconditionally, so a value you set beforehand is lost. Set your own afterwards. ## Layer two: the live check A green `apply` means Terraform's API calls completed. It does not promise that the endpoint in the module is ready to answer. The check therefore has to poll: - `http_helper.HTTPGetWithRetryContext(t, ctx, url, tlsConfig, 200, "ok", 30, 5*time.Second)` repeats until status and body match exactly, and fails the test if retries run out. - `http_helper.HTTPGetWithRetryWithCustomValidationContext` takes your own function for a looser check. - `retry.DoWithRetryContext(t, ctx, description, maxRetries, sleep, action)` wraps any check you can write. `retry.DoWithRetryContext` repeats on **any** error unless the action returns `retry.FatalError`, which stops the loop immediately. That is the opposite default from layer one, where only listed patterns repeat. So custom checks need to mark what cannot heal themselves. ## Working out the budget `maxRetries` counts retries, not attempts, and the loop sleeps after each failure, including the last. So: 1. `maxRetries = 30` means up to **31 attempts**. 2. With a 5-second sleep, that is 155 seconds of sleeping, plus the time each attempt takes (`http_helper` uses a 10-second timeout per request). 3. A slow, unanswered endpoint can therefore take far longer than the 2.5 minutes the arguments suggest. 4. That total sits inside `go test -timeout`, alongside the apply and the destroy. If the retries use up the whole budget, Go aborts the binary and the deferred destroy never runs, so an over-generous retry can cause a leak. Contexts help you cap the total, but the two helpers treat them differently: - **`retry.DoWithRetryContext`** checks the context before each attempt and during each sleep, so a deadline ends the loop. - **`http_helper.HTTPGetWithRetryContext`** passes the context to each request, not to its retry loop. Once the context expires, each request fails, and the loop still works through its remaining retries and sleeps. For a hard cap on the check, the first helper is the one to use, wrapped in a context from `context.WithTimeout`. Keep that context away from the deferred destroy, because an expired context stops `terraform destroy` from starting. ## What should not be retried Retrying is only right when the system's contract says the answer will arrive later. These failures will not heal: - **A validation or configuration error** from Terraform. Keep such messages out of `RetryableTerraformErrors`; a pattern like `.*Error.*` would repeat every failure. - **An endpoint that answers with the wrong content.** It is up and it is wrong, so return `retry.FatalError` rather than spending the budget. - **An expired credential.** More attempts only waste time. A retry that only ever succeeds on attempt 25 is also information. It may mean the module lacks a real wait, such as a health check it should declare, so look at the count in the logs rather than simply raising it. Terratest logs each retry, which makes that review possible.

  • In Terratest, how do you stop `retry.DoWithRetryContext` spending its budget on an error that cannot heal?
    Return `retry.FatalError{Underlying: err}` from the action. The loop detects that type, logs it and returns immediately, and `DoWithRetryContext` then fails the test. Any other error is treated as retryable, so without this a check for a permanent condition repeats until `maxRetries` runs out. The `E` variant returns the same error to you instead.
  • A Terratest suite sets `MaxRetries = 10`, then wraps the options with `WithDefaultRetryableErrors`. How many retries does it get?
    Three. `WithDefaultRetryableErrors` clones the options and assigns `MaxRetries = 3` and a 5-second `TimeBetweenRetries` without checking what was there. Any custom patterns already in `RetryableTerraformErrors` are kept and merged. To use 10, set the field on the returned copy after the call.

  • WithDefaultRetryableErrors repeats every Terraform command that fails, whatever the error.
  • A plain terraform.Options value already retries transient errors by default.
  • A successful terraform apply means the endpoint can serve traffic immediately.
  • A very large retry count is a safe fix for any flaky infrastructure check.
  • retry.DoWithRetryContext gives up by itself on errors that look permanent.
  • A MaxRetries value set before WithDefaultRetryableErrors is kept by that call.