An AWS CloudFormation stack update fails and the stack ends up in UPDATE_ROLLBACK_FAILED. What does that status actually mean, and how do you get the stack back to a state where you can deploy again?

## How a stack gets there A CloudFormation stack update is a two-phase operation with an automatic safety net. Phase one applies the new template. If any resource fails, CloudFormation stops and enters `UPDATE_ROLLBACK_IN_PROGRESS`, walking back every change it already made so the stack returns to the template it had before. The happy ending of that path is `UPDATE_ROLLBACK_COMPLETE` — a stack that is exactly where it started, updatable again. `UPDATE_ROLLBACK_FAILED` is what you get when the safety net itself tears. A rollback step is just another API call against a real resource, and it can fail for ordinary reasons: - Somebody deleted or renamed a stack-managed resource in the console, so there is nothing to restore. - A resource cannot be returned to its old configuration — an S3 bucket that must be emptied first, a security group still referenced by something outside the stack, a subnet with an ENI left in it. - The IAM identity or stack role executing the update lost a permission partway through, or a policy changed under it. - A resource that waits for a signal or a stabilisation timeout never reports back, and the rollback times out too. - A nested child stack failed its own rollback, which fails the parent's. The important operational point: the workload is still running. `UPDATE_ROLLBACK_FAILED` is not an outage by itself. It is a stack that is half-way between two templates and that CloudFormation refuses to touch further, which means every subsequent deploy is blocked until it is cleared. ## What CloudFormation will accept in that state Exactly two things: `ContinueUpdateRollback`, and `DeleteStack`. `update-stack` is rejected. `cancel-update-stack` is not applicable — that only works while an update is still in progress. Creating a change set against the stack does not help either, because you cannot execute it. ## The recovery, in order **1. Find the actual failure.** `describe-stack-events` returns newest first; scroll back to the first `UPDATE_FAILED` or `UPDATE_ROLLBACK_FAILED` event and read `ResourceStatusReason`. Later events are usually cascades from that one. This is the whole diagnosis — the reason string almost always names the problem ("The bucket you tried to delete is not empty", "resource does not exist", an AccessDenied). **2. Repair the blocker by hand.** This is the part that feels wrong to engineers who believe infrastructure should never be touched manually, and it is nonetheless the correct move. If a resource was deleted, recreate something CloudFormation can find at that identifier. If a bucket is not empty, empty it. If a permission was removed, put it back. You are not fixing the infrastructure; you are removing the obstacle to the rollback step. **3. Continue the rollback.** ```bash aws cloudformation continue-update-rollback --stack-name payments-prod ``` CloudFormation retries the remaining rollback steps from where it stopped. If it succeeds you reach `UPDATE_ROLLBACK_COMPLETE` and the stack is deployable again. **4. Skip only what cannot be recovered.** When a resource genuinely cannot be restored, name its logical ID in `--resources-to-skip` (nested-stack resources are addressed as `NestedStackName.LogicalId`): ```bash aws cloudformation continue-update-rollback --stack-name payments-prod \ --resources-to-skip AppDatabase LoggingStack.LogGroup ``` CloudFormation marks those resources as if their rollback had succeeded and finishes. That unblocks the stack, and it also means CloudFormation's recorded state for those resources no longer matches what exists. You have deliberately traded accuracy for a working stack. The debt is real: the next update computes its diff from a record you know is wrong. Pay it immediately, either by making the live resource match what the stack believes or by updating the template to describe what is actually there. **5. If none of that works,** the remaining option is deleting the stack and recreating it — viable in a lower environment, rarely acceptable in production unless the stateful resources are protected by retaining policies and can be brought back into a fresh stack. ## Reducing how often this happens - Review a change set before executing, so fewer updates fail in the first place. - Stop people editing stack-managed resources by hand; that single habit causes most unrecoverable rollbacks. - Give the stack its own role with `--role-arn` so the rollback does not depend on whichever human or pipeline credential started the update still having permissions. - Since 2021 you can pass `--disable-rollback` on `update-stack`. The failed update then stops in `UPDATE_FAILED` with the partially updated resources left alone, so you can inspect them; from there you either continue rolling back or call `rollback-stack` when you are ready. It converts a panicked automatic rollback into a decision you make with evidence. - Attaching CloudWatch alarms as rollback triggers via `--rollback-configuration`, with `MonitoringTimeInMinutes`, makes CloudFormation watch for a set period after the update and roll back if an alarm fires — worth knowing, and worth remembering that it adds another way a rollback can start.