Why do in-flight executions of a days-long hosted workflow fail after you deploy a new version, and how do you ship workflow changes safely?
answer
- executions outlive deploys
- old history meets new code
- steps calling the latest function
- a version gate per change
- drain the old, route the new
basics
~20 sExecutions started before the deploy resume against changed code: replayed workflow code requests different steps than their history recorded, or steps call the newest function version with old-shaped input. Gate changes by version, pin step targets, and drain old executions.
solid answer
~50 sA document pipeline whose executions wait days for review always has executions **in flight** during a deploy, and they resume against whatever is deployed then. Two failures follow. In **code-first durable execution**, the engine replays old histories through new code; if the change adds, removes or reorders steps, the code requests a different sequence than the history recorded and the execution stops with a **non-determinism error**. In a **definition-based** engine an execution usually keeps the definition it started with, but if its steps call functions by unversioned name, the remaining steps hit **new function code with old-shaped input**. To ship safely: put each code-first change behind a **version gate** that keeps old executions on the old path; pin step targets to **immutable function versions or aliases** per workflow version; keep inputs backward compatible; **run old and new side by side** until old executions drain; and replay recorded histories against new code before deploying.
code
pseudocode · 11 linesworkflow publishDocument(documentId):
text = step("extractText@v7", documentId) # pinned function version
# added in this release; the engine records which branch was taken
v = engine.versionGate("add-pii-scan", oldest = 1, newest = 2)
if v >= 2:
text = step("scanForPii@v1", text)
decision = waitForCallback("review", documentId)
if decision.approved:
step("publish@v4", documentId, text)go deeper
Recall that a workflow execution can outlive many deploys, so code changes reach executions that started under the old version.
Explain the two failure modes: replay through changed workflow code, and steps calling the newest function version with old-shaped input.
Diagnose from the execution history and the deploy time, then ship with version gates, pinned function versions, compatible inputs and a drain of old executions.
Set the team's release rules for long-running workflows: replay tests in CI, version-gate review, and how long old versions stay deployed.
## Why long executions collide with deploys A hosted workflow execution can last as long as its slowest wait. In a document pipeline with a human review step, that is days, and the team deploys several times in those days. Every deploy therefore lands on **executions in flight**: started under the old code, paused at a review or a timer, and resumed after the deploy under whatever is current then. The symptom is specific: executions **started after** the deploy are fine, while executions **started before** it fail or misbehave when they resume. That pattern points at versioning, not at a plain bug in the new code. ## Failure mode one: replay against changed code In **code-first durable execution**, the engine rebuilds a paused execution by replaying the workflow code against its recorded history. A change that alters the sequence of steps, such as adding a scan step before review, removing a notification or swapping two steps, makes the new code request something different from what the history says happened. The engine detects the mismatch and raises a **non-determinism error**; the execution is stuck until the code is fixed or the execution is reset. Changes that do not alter the sequence of step requests, such as fixing a bug inside a step's own function, are generally safe for replay. Whether they are safe for the business is a separate question. ## Failure mode two: steps that call whatever is latest In a **definition-based** engine an execution typically runs to completion on the definition it started with, so the workflow structure is stable. The trap is in the steps: if a step names its function without a version, it calls the **newest deployed code**. An execution that extracted text with the old function now calls a new publish function expecting a field the old extraction never produced. Nothing in the engine reports a versioning problem; the step fails with an ordinary error, or worse, succeeds with wrong data. | | Code-first durable execution | Definition-based state machine | |---|---|---| | What changes under an in-flight execution | The workflow code it is replayed through | The functions its remaining steps call | | Typical symptom | Non-determinism error on resume | Step errors or wrong output from contract mismatch | | Main guard | Version gates in workflow code | Pinning step targets to function versions or aliases | ## Shipping workflow changes safely 1. **Gate each structural change.** In code-first engines, wrap a new or changed step in a version check that records which branch an execution took. Executions that passed that point before the deploy replay the old branch; new ones take the new branch. 2. **Pin step targets.** Have each workflow version call **immutable function versions** or an alias per workflow version, so old executions keep calling code that understands their data. 3. **Keep inputs compatible.** Add fields before you require them and stop requiring them before you remove them, so a step can accept both the old and new shapes during the overlap. 4. **Run old and new side by side.** Route new executions to the new version while old ones finish on the old, and keep the old version deployed until the count of its in-flight executions reaches zero. 5. **Test against real histories.** Replay a sample of recorded production histories through the new code in CI; a non-determinism error fails the build instead of a customer's document. 6. **Clean up later.** Remove a version gate and the old function versions only once no execution that could replay through them remains. ## Reading the execution history to confirm When an execution fails after a deploy, its **execution history** is the evidence: - Find the last event that succeeded and the event that failed; compare their timestamps with the deploy time. - For a non-determinism error, the engine usually reports which step the code requested and which the history expected. - For a contract mismatch, compare the failed step's input with what the new function expects, and check which function version it invoked. - Count in-flight executions by version before every deploy; the count tells you how long the old path must stay. The cost of all this is real but bounded: a few lines of version checks, old function versions kept for days, and a replay test in the pipeline. The alternative, cancelling and restarting in-flight executions after each deploy, throws away human review work and does not scale with deploy frequency.
- When is it safe to delete a version gate from durable workflow code?Only when no execution that could replay through the old branch remains: every execution that started before the gate has completed, and any retention window during which the engine might replay or reset completed executions has passed. Check the count of open executions by version rather than guessing from the calendar, then remove the gate in its own release.
- How do you catch a breaking workflow change before it reaches production?Export a sample of recent execution histories from production, including executions paused mid-flow, and replay them through the new workflow code in CI. A non-determinism error fails the build. For definition-based engines, the equivalent is a contract test: feed recorded step inputs from old executions into the new function versions and check they still succeed.
- Why not just cancel and restart in-flight executions after each deploy?Restarting re-runs work that already happened, such as extraction already billed and reviews a person already did, and repeats side effects unless every step is idempotent. With executions lasting days and several deploys a week, it means some documents never finish. It is a last resort for a broken release, not a release process.
- Deploying a new workflow version safely migrates running executions onto it automatically.
- Reordering two steps is harmless as long as both of them still run.
- Pinning the workflow definition alone pins behaviour, even if steps call unversioned functions.
- Cancelling and restarting every in-flight execution after a deploy costs nothing.
- Old function versions can be deleted right after a deploy because new executions use the new ones.