A failed pipeline run leaves the Terraform state object in your S3 backend truncated and unusable. How does bucket versioning let you recover, and what do you check after restoring?

## Why versioning is the whole safety net Terraform state in an object-storage backend is one key that is overwritten in place on every apply. A crashed process, a truncated upload, a mistaken `aws s3 cp`, or a state operation gone wrong replaces the authoritative record of your production estate with garbage — and Terraform cannot rebuild it, because it has no way to enumerate which cloud objects were "its". Versioning on the bucket converts that from a catastrophe into an inconvenience, which is why it belongs in the bucket's definition from day one rather than after the first incident. The equivalents exist elsewhere: object versioning on GCS buckets, and blob versioning or soft delete on the Azure storage account used by the `azurerm` backend. Pair versioning with a noncurrent-version lifecycle rule that keeps old versions long enough to survive a slow-to-notice corruption — days, not hours — and with encryption at rest and a policy that restricts who can write or delete the key at all. ## The recovery procedure **1. Stop the bleeding.** Freeze the pipeline. Any further apply against a corrupt state can create duplicates or destroy real resources, and every write buries the good version deeper. **2. Enumerate versions.** ```bash aws s3api list-object-versions \ --bucket acme-tfstate \ --prefix prod/network/terraform.tfstate ``` You get version IDs with timestamps. The most recent is the broken one. **3. Fetch and inspect candidates.** Download each candidate with `aws s3api get-object --version-id ...` and check two things inside the JSON: it parses at all, and the `serial` and `lineage` fields look right. Terraform increments `serial` on each write and keeps `lineage` constant for the life of a state; a candidate with a different lineage belongs to a different state entirely and must not be restored here. **4. Restore.** Either copy the chosen version back over the key, making it the new current version, or use `terraform state push` with the downloaded file. Note that `state push` performs safety checks — it refuses when the pushed serial is lower than the remote one or the lineage differs, and `-force` bypasses those checks. Copying the object directly in S3 bypasses the checks entirely, which is faster and blinder; either way the verification step below is what actually protects you. **5. Verify with a plan.** `terraform plan` is the audit: - *No changes* — the restored snapshot matches reality; you are done. - *Creates* — the failed run genuinely created resources after the snapshot was taken. They exist in the cloud but not in state, so applying would build duplicates. Adopt them into state instead of creating them. - *Destroys* — the restored state is ahead of reality, or the failed run deleted things; check each one before letting an apply act on it. - *In-place updates* — usually harmless drift in cached attributes that the next refresh reconciles. **6. Clear any leftover lock.** A process that died mid-apply may not have released the backend lock, so the first recovery run can fail to acquire it. That is a separate mechanism with its own safety rules — never break a lock you have not confirmed is stale. ## Judgment, not just commands The hard part is step 5. Restoring an old state is not "undo": infrastructure kept moving while state did not, so what you have restored is a snapshot of the world as of a moment that has passed. Reconciling the difference between the snapshot and reality is the actual recovery, and it must be done by reading the plan resource by resource rather than by trusting the exit code. The safest posture is to restore the *newest* version that is intact, minimising the window you have to reconcile by hand. It is also worth asking why the truncated write happened. Common causes: a CI runner killed mid-apply by a job timeout or a spot reclaim; an engineer running a state operation against the wrong key; a network failure during upload. The first is best fixed by making apply jobs uninterruptible and generously timed, the second by tightening who can write to state at all. ## Prevention that costs nothing Versioning on. Encryption on. A bucket policy that denies deletes to everything except a break-glass role. One state key per root module so a bad operation cannot take out several estates. And treat the state bucket itself as infrastructure that is created once and never casually re-provisioned. ## How to say it in an interview "Versioning means the previous state object is still there. List versions, take the newest intact one, restore it, and then run a plan — creates in that plan are resources the failed run made that the snapshot predates, and those get imported, not created."