A Terratest test of a VPC module fails its endpoint check right after apply but passes on rerun: where do retries belong, and what should not be retried?
answer
- two layers, two kinds of failure
- apply finished does not mean ready
- only listed patterns repeat
- one error type ends the loop
- the budget includes the destroy
basics
~10 sRetry at two layers: terraform.WithDefaultRetryableErrors re-runs a Terraform command on listed transient errors, and http_helper or retry.DoWithRetryContext polls the live check. Deterministic failures should stop at once, and retries must fit the test timeout.
solid answer
~40 sTwo layers, because they fail differently. A successful `apply` says the API calls finished, not that the endpoint answers yet, so I poll with `http_helper.HTTPGetWithRetryContext` or `retry.DoWithRetryContext`, which repeats on any error except `retry.FatalError`. For Terraform commands, `terraform.WithDefaultRetryableErrors` sets `MaxRetries` to 3 and `TimeBetweenRetries` to 5 seconds, and repeats only failures matching its regex list, such as `Provider produced inconsistent result after apply`; anything else fails immediately. A bare `terraform.Options` retries nothing. What should not repeat is a deterministic failure: a validation error, or an endpoint answering with the wrong content. Broad patterns and huge counts turn those into slow failures and can hide a flaky defect. Everything must also fit inside `go test -timeout`, with room left for the destroy.
code
go · 15 linesurl := terraform.OutputContext(t, ctx, options, "endpoint_url")
retry.DoWithRetryContext(t, ctx, "GET "+url, 30, 5*time.Second, func() (string, error) {
status, body, err := http_helper.HTTPGetContextE(t, ctx, url, &tls.Config{})
if err != nil {
return "", err // not answering yet: retry
}
if status != http.StatusOK {
return "", fmt.Errorf("status %d", status) // not ready yet: retry
}
if body != expected {
return "", retry.FatalError{Underlying: fmt.Errorf("wrong body: %s", body)} // will not heal
}
return body, nil
})go deeper
Recall that a finished apply does not mean a service is ready, and that Terratest has retry helpers for HTTP checks and for Terraform commands.
Explain the two defaults: Terraform commands repeat only on listed patterns, while retry.DoWithRetryContext repeats on anything except retry.FatalError. Know that a bare Options value does not retry.
Work out the retry budget against go test -timeout, including the destroy, and show how an over-generous retry can cause a leak. Say what you would mark as fatal and why.
Set a team standard: shared retry settings, a review bar for new patterns, and retry counts watched as a signal, so that accumulating retries does not quietly hide unreliable modules.
Many cloud APIs are **eventually consistent**: a change they accept can take time to become visible everywhere. Live-account tests meet this often, and Terratest offers separate tools for the two places it shows up. This answer follows Terratest 1.0.1. ## Layer one: the Terraform command The command itself sometimes fails for reasons that heal on their own: a plugin download hits a network error, or a provider reads back a value that has not propagated yet. `terraform.Options` controls this with three fields, and `terraform.WithDefaultRetryableErrors` fills them in. | Field | Bare `terraform.Options` | After `WithDefaultRetryableErrors` | |---|---|---| | `RetryableTerraformErrors` | Empty | Regex list of known transient errors | | `MaxRetries` | 0 | 3 | | `TimeBetweenRetries` | 0 | 5 seconds | The behaviour has these details: - **Matching decides.** Each failure's output and error text are checked against the regexes. A match waits and repeats; no match is wrapped in a `retry.FatalError` and returned at once. - **What is listed.** The defaults include registry and plugin errors, connection resets and `Provider produced inconsistent result after apply`, which a source comment describes as usually an eventual consistency issue. - **It covers every Terraform command**, because all of them go through `RunTerraformCommandContextE`, so `init` and `destroy` benefit as well as `apply`. - **It returns a copy.** Use the returned value, because your original options are unchanged. - **It overwrites two fields.** `MaxRetries` and `TimeBetweenRetries` are set unconditionally, so a value you set beforehand is lost. Set your own afterwards. ## Layer two: the live check A green `apply` means Terraform's API calls completed. It does not promise that the endpoint in the module is ready to answer. The check therefore has to poll: - `http_helper.HTTPGetWithRetryContext(t, ctx, url, tlsConfig, 200, "ok", 30, 5*time.Second)` repeats until status and body match exactly, and fails the test if retries run out. - `http_helper.HTTPGetWithRetryWithCustomValidationContext` takes your own function for a looser check. - `retry.DoWithRetryContext(t, ctx, description, maxRetries, sleep, action)` wraps any check you can write. `retry.DoWithRetryContext` repeats on **any** error unless the action returns `retry.FatalError`, which stops the loop immediately. That is the opposite default from layer one, where only listed patterns repeat. So custom checks need to mark what cannot heal themselves. ## Working out the budget `maxRetries` counts retries, not attempts, and the loop sleeps after each failure, including the last. So: 1. `maxRetries = 30` means up to **31 attempts**. 2. With a 5-second sleep, that is 155 seconds of sleeping, plus the time each attempt takes (`http_helper` uses a 10-second timeout per request). 3. A slow, unanswered endpoint can therefore take far longer than the 2.5 minutes the arguments suggest. 4. That total sits inside `go test -timeout`, alongside the apply and the destroy. If the retries use up the whole budget, Go aborts the binary and the deferred destroy never runs, so an over-generous retry can cause a leak. Contexts help you cap the total, but the two helpers treat them differently: - **`retry.DoWithRetryContext`** checks the context before each attempt and during each sleep, so a deadline ends the loop. - **`http_helper.HTTPGetWithRetryContext`** passes the context to each request, not to its retry loop. Once the context expires, each request fails, and the loop still works through its remaining retries and sleeps. For a hard cap on the check, the first helper is the one to use, wrapped in a context from `context.WithTimeout`. Keep that context away from the deferred destroy, because an expired context stops `terraform destroy` from starting. ## What should not be retried Retrying is only right when the system's contract says the answer will arrive later. These failures will not heal: - **A validation or configuration error** from Terraform. Keep such messages out of `RetryableTerraformErrors`; a pattern like `.*Error.*` would repeat every failure. - **An endpoint that answers with the wrong content.** It is up and it is wrong, so return `retry.FatalError` rather than spending the budget. - **An expired credential.** More attempts only waste time. A retry that only ever succeeds on attempt 25 is also information. It may mean the module lacks a real wait, such as a health check it should declare, so look at the count in the logs rather than simply raising it. Terratest logs each retry, which makes that review possible.
- In Terratest, how do you stop `retry.DoWithRetryContext` spending its budget on an error that cannot heal?Return `retry.FatalError{Underlying: err}` from the action. The loop detects that type, logs it and returns immediately, and `DoWithRetryContext` then fails the test. Any other error is treated as retryable, so without this a check for a permanent condition repeats until `maxRetries` runs out. The `E` variant returns the same error to you instead.
- A Terratest suite sets `MaxRetries = 10`, then wraps the options with `WithDefaultRetryableErrors`. How many retries does it get?Three. `WithDefaultRetryableErrors` clones the options and assigns `MaxRetries = 3` and a 5-second `TimeBetweenRetries` without checking what was there. Any custom patterns already in `RetryableTerraformErrors` are kept and merged. To use 10, set the field on the returned copy after the call.
- WithDefaultRetryableErrors repeats every Terraform command that fails, whatever the error.
- A plain terraform.Options value already retries transient errors by default.
- A successful terraform apply means the endpoint can serve traffic immediately.
- A very large retry count is a safe fix for any flaky infrastructure check.
- retry.DoWithRetryContext gives up by itself on errors that look permanent.
- A MaxRetries value set before WithDefaultRetryableErrors is kept by that call.