After wrangler deploy changed a Cloudflare Container's image, some requests still reached the old image for minutes and a few failed. Why?
answer
- the Worker goes live first
- not one transaction
- steps of 10 then 100
- SIGTERM, then a long drain
- --containers-rollout
basics
~20 swrangler deploy activates the new Worker first, then builds the image and rolls instances gradually, about 10% then 100%. Old instances drain up to 15 minutes after SIGTERM, so new code meets old images and cold starts.
solid answer
~40 sFor a default-policy Container application, `wrangler deploy` uploads and activates the new Worker version, then builds and pushes the image, then starts a rollout; the steps are not transactional, and success means the rollout started. With `max_instances` of 2 or more, `rollout_step_percentage` defaults to `[10, 100]`. Each replaced instance gets `SIGTERM`, up to 15 minutes to exit, then `SIGKILL`, and its successor cold-starts. So for minutes the new Worker talks to old images: a changed request shape breaks there, and requests waiting on a restarting instance time out. Fixes: keep Worker and image compatible across the window, handle `SIGTERM`, keep startup fast, use `--containers-rollout=immediate` for unavoidable breaks, and `rollout_active_grace_period` to spare young sessions. `wrangler versions upload` publishes no image at all.
code
json · 11 lines{
"containers": [
{
"class_name": "Converter",
"image": "./container/Dockerfile",
"max_instances": 10,
"rollout_active_grace_period": 300,
"rollout_step_percentage": [10, 50, 100]
}
]
}go deeper
Recall that wrangler deploy builds and pushes a Dockerfile image and that container instances update after the Worker, not with it.
Explain the deploy order, the default rollout steps, and the SIGTERM, 15-minute drain, SIGKILL sequence for each replaced instance.
Diagnose mixed-version failures and startup timeouts, and fix them with compatible changes, SIGTERM handling, readiness checks and the right rollout flag.
Set the team's release rule for container changes: expand-then-contract contracts, when immediate is allowed, and where new images are tested before production.
## What `wrangler deploy` actually does For an existing Container application on the `default` scheduling policy, when the effective container configuration changed, `wrangler deploy` runs three steps in order: 1. Uploads and **activates the new Worker version**, including the Durable Object class code. 2. Builds and pushes the image when `image` is a Dockerfile path, or resolves the registry reference. 3. Starts a **rollout** that applies the new image or instance settings to running instances. The steps are **not transactional**. The Worker is live before the image exists, so a failed build or push leaves new Worker code in front of old containers, and a successful deploy only means the rollout started. The first deploy creates the application directly, and a deploy with no effective container change starts no rollout. ## Why old images keep answering The rollout is gradual by default. `rollout_step_percentage` defaults to a single `100` step when `max_instances` is omitted or below 2, and to `[10, 100]` otherwise: about 10% of instances (at least one) first, then all of them. Each instance the rollout selects is replaced on its own schedule: 1. **Grace period.** With `rollout_active_grace_period` above 0, instances connected to their Durable Object for fewer seconds than that are skipped for now; the default is 0. 2. **Signal.** The main process receives `SIGTERM`. 3. **Drain.** The process has up to 15 minutes to exit. 4. **Force.** If it is still running, it receives `SIGKILL`. 5. **Replace.** A new instance starts from the new image, with a cold start of seconds. The fleet never flips in one moment. For minutes, sometimes longer when processes use their full drain, the new Worker reaches some instances on the old image and some on the new one. ## Why a few requests failed - **Contract mismatch.** The new Worker sends a path, field or header the old image does not understand, and the old instance answers with an error. - **Startup gap.** A request for an instance being replaced waits for the new process to boot; a client or Worker timeout shorter than the startup turns that wait into a failure. - **Lost work.** A process that exits at once on `SIGTERM` drops the document it was converting, and one that ignores the signal is killed after 15 minutes anyway. - **Half-finished deploy.** If the image build failed after the Worker went live, requests that need the new container behaviour can fail until the next deploy. ## The controls | Control | Effect | | --- | --- | | No flag (gradual) | Follows `rollout_step_percentage` | | `--containers-rollout=immediate` | One step to 100%: the shortest mixed window, never zero; the grace period still applies | | `--containers-rollout=none` | Deploys Worker code only; images and running instances stay unchanged | | `rollout_active_grace_period` | Spares instances whose Durable Object connected recently | | `wrangler versions upload` | Uploads a Worker version only; publishes no image and starts no rollout | `--containers-rollout` belongs to `wrangler deploy` alone. Workers Builds runs `npx wrangler versions upload` on non-production branches by default, so a feature-branch build never updates the image, and Version URLs are not generated for Workers that implement Durable Objects. A staging Worker or Wrangler environment with a full deploy is the way to try a new image before production. ## Fixing the incident - Make the change **compatible across the window**: first ship an image that accepts both the old and the new request shape, then the Worker that sends the new one, then remove the old shape. - **Handle `SIGTERM`** in the converter: stop accepting work, finish or checkpoint the current document, then exit. - Keep startup fast, check readiness before forwarding, and give clients timeouts and retries longer than a cold start. - Reserve `--containers-rollout=immediate` for breaking changes you cannot split, knowing it shortens the window without closing it. - Watch progress with `wrangler containers list`, `wrangler containers instances <APPLICATION_ID>` or the dashboard, rather than treating a green deploy as done. ## The `durable_object` policy is different Applications on the `durable_object` scheduling policy have no rollouts. The image map is uploaded with the Worker version; a running instance keeps the image it started with until your code stops it. To upgrade, compare `ctx.container.inspect()`'s `image` with `ctx.container.images.base` and, when they differ, `destroy()` and `start()` again. During a gradual Workers deployment, different Durable Objects can see different image maps until it completes.
- When would you deploy with --containers-rollout=none?When only Worker code changed and you must not touch running containers, for example a routing fix during a busy period, or when the deploy machine has no Docker for a Dockerfile image. It activates the Worker and skips building, pushing and rolling out the image; ship the image change in a later full deploy.
- How do you test a new container image before production if feature-branch builds cannot?Non-production Workers Builds run `wrangler versions upload`, which publishes no image, and Durable Object Workers get no Version URLs. Use local `wrangler dev` with Docker, or a separate staging Worker or Wrangler environment whose build runs a full `wrangler deploy --env staging` with its own Container application.
- What does rollout_active_grace_period protect, and what does it not?It keeps a rollout from replacing an instance whose Durable Object connected fewer than that many seconds ago, so short sessions can finish. It applies in every mode, including immediate. It does not delay rollout steps, extend the 15-minute `SIGTERM` drain, or protect instances that are already older than the window.
- Believes wrangler deploy swaps the Worker and every container atomically.
- Reads a successful deploy as proof that all instances run the new image.
- Thinks --containers-rollout=immediate removes the mixed-version window entirely.
- Expects wrangler versions upload to publish the new container image.
- Assumes the platform kills replaced instances instantly with no SIGTERM drain.
Assumes Wrangler 4.149.0 on the platform as of 2026-10-10: --containers-rollout (gradual, immediate, none), rollout_step_percentage defaults and the durable_object policy's lack of rollouts.