During terraform apply an instance is created successfully but its remote-exec provisioner fails halfway through. What state is that resource left in, and what does the next apply do?

## What "tainted" means Tainted is a per-instance flag recorded in Terraform state. It says: this object exists in the real world, but Terraform believes it is not in a usable condition, so it should be destroyed and recreated rather than reused. It is not a separate object and not a deletion — the cloud resource is still running and still costing money. A creation-time provisioner failure is the *automatic* way a resource becomes tainted. Terraform's reasoning is straightforward: you declared the provisioner as part of creating this resource, so if it did not complete, the resource was not fully created. ## The apply sequence 1. The provider creates the object and returns its attributes; Terraform writes them to state immediately. 2. Terraform runs the resource's provisioners, in declaration order. 3. A provisioner exits non-zero. Terraform records the error, marks the instance tainted, writes state, and fails the apply with a non-zero exit code. Step 1 matters: state is written *before* provisioning, which is why the object is not orphaned. If Terraform had not recorded it, you would be left with an untracked instance running in the account. ## What the next plan shows ``` # aws_instance.web is tainted, so must be replaced -/+ resource "aws_instance" "web" { ``` The `-/+` marker means destroy-then-create. On apply, the tainted object is destroyed, a fresh one is created, and the full provisioner sequence runs again from the top. There is no partial resume — provisioners have no checkpointing. ## on_failure Every provisioner block accepts `on_failure`, with two values: - `fail` (the default) — error out and taint the resource. - `continue` — log the error, ignore it, and carry on with the rest of the apply. The resource is **not** tainted. ```hcl provisioner "remote-exec" { inline = ["/opt/optional-metrics-agent.sh"] on_failure = continue } ``` Note it is a bare keyword, not a quoted string. Use `continue` only for genuinely optional side work; using it to paper over a flaky bootstrap gives you machines that report success while being half-configured — strictly worse than a loud failure. ## Destroy-time failures behave differently A `when = destroy` provisioner that fails does not taint anything — instead Terraform errors and the resource stays in state, and the provisioner will be attempted again on the next apply that tries to destroy it. That is why the docs warn destroy-time provisioners must be safe to run more than once. ## Clearing or forcing the flag manually Since Terraform 0.15.2 the recommended way to force replacement is the plan option `-replace=ADDRESS` rather than the older `terraform taint` command, which is deprecated. To go the other way and tell Terraform the object is actually fine, `terraform untaint ADDRESS` clears the flag — appropriate only when you have verified by hand that the machine is correctly configured, for example because the provisioner failed on its very last, harmless step. ## Why this behaviour argues against provisioners Consider the operational picture. A transient DNS hiccup or a package mirror timeout during bootstrap now costs a full instance replacement — and if that instance is in an autoscaling-adjacent or stateful position, the blast radius is real. Compare with `user_data`: if a cloud-init script fails, the instance still exists and boots, Terraform's apply succeeds, and you debug the box by reading `/var/log/cloud-init-output.log`. The failure is contained to the machine rather than escalating into an infrastructure change. The deeper problem is the half-executed script. Terraform knows only that the command exited non-zero; it has no idea which of the eight lines in your `inline` list ran. If someone reaches for `on_failure = continue` or `terraform untaint` to get the pipeline green, the fleet quietly acquires a machine nobody can reason about. Destroy-and-recreate is the safe default precisely because it refuses to reason about partial state. ## What to say in an interview Name the flag (tainted), state that the real object still exists and is billed, describe the destroy/recreate plan, mention `on_failure` and its two values, and close with the judgment: the harshness of this failure mode is one of the strongest practical arguments for putting bootstrapping in the image or in `user_data` instead.