Should a platform team standardise multi-step function jobs on a hosted workflow engine or keep coordination in application code, and what decides it?
answer
- what home-grown coordination keeps re-solving
- visibility of every single run
- price per step at volume
- the definition language as lock-in
- a standard per class of workflow
basics
~20 sStandardise on a hosted engine for long, branching, waiting or fan-out workflows where per-execution history and built-in retries pay off; keep short, high-volume chains in code. Weigh per-step cost, definition-language lock-in and team discipline, not features alone.
solid answer
~50 sIt is a standard per **class of workflow**, not one answer for everything. A hosted engine earns its place where jobs run long, wait on people or callbacks, fan out, and need an **auditable per-execution history**: it replaces the status tables, retry loops and stuck-job scripts every team otherwise rebuilds. It adds standing costs: **per-step pricing** that punishes short, high-volume flows; a **definition language** or client library specific to that engine, which is lock-in; payload and history caps; and, for code-first engines, **determinism discipline** on every developer. Coordination in code (a queue between functions plus a status record) stays cheaper and simpler for two- or three-step chains at high volume. I would decide from evidence: incidents caused by lost or stuck jobs, how many teams maintain their own coordination, and a step-count times volume cost estimate, then publish a golden path with both options and the criteria that pick between them.
go deeper
Recall the two options: a hosted engine that keeps workflow state and history, or functions linked by queues with state kept in your own database.
Explain what an engine provides out of the box (durable state, retries, waits, history) and what coordination in code has to rebuild.
Show the operating evidence for each option: stuck-job incidents, history-based debugging, per-step cost at volume, and payload and history caps.
Own the standard: a per-class rule backed by a cost estimate and incident data, a golden path, versioning rules, and a planned exit cost.
## What the decision really is Every organisation that runs functions ends up with multi-step jobs: extract then review then publish, charge then provision then notify. The steps have to be coordinated somewhere. The choice a platform lead faces is whether that coordination becomes a **shared standard on a hosted workflow engine** or stays **inside each team's application code**, usually a queue between functions plus a status record in a database and some retry logic. This is not the choreography versus orchestration question. Both options here are orchestration; the question is who runs the orchestrator and who pays for it. ## What a hosted engine buys - **Durable state for free.** Executions survive crashes, deploys and days-long waits without each team designing a status table. - **Built-in retries, timeouts and callback waits**, configured instead of hand-written and re-tested by each team. - **Per-execution history.** An operator can open one document's run and see every step, input, output and error. In-code coordination usually reconstructs this from logs. - **Fan-out with a concurrency cap** and a fan-in, without building a job tracker. - **A shared operating model**: one set of dashboards, alarms and runbooks across teams. ## What it costs, every month - **Per-step pricing.** Many engines bill per state transition or per step. A long workflow with few steps is cheap; a short chain run millions of times can cost more in engine charges than in function compute. - **Lock-in.** A definition language or durable-execution client library is specific to its engine. Moving providers means rewriting workflows, not redeploying them. - **Hard caps.** Most engines cap the data passed between steps and, often, the size of an execution's history, which pushes designs toward storage references and batching. - **Discipline.** Code-first engines require deterministic workflow code; definition-based engines push complex logic into an awkward language. Either way, teams need training and review. - **Versioning work.** Long-running executions outlive deploys, so every structural change needs version gates or pinned function versions. - **A platform dependency.** The engine's outages and limits become everyone's outages and limits. ## Comparing the options | Concern | Definition-based hosted engine | Code-first durable engine | Coordination in application code | |---|---|---|---| | Long waits and callbacks | Built in | Built in | Status table plus a scheduler | | Run visibility | Execution history with a visual graph | Execution history and replay | Logs and status rows | | Cost shape | Per step, or duration on some types | Per action or self-run hosts | Function compute plus queue and storage | | Flexibility of logic | Limited to the definition language | A general-purpose language | A general-purpose language | | Lock-in | High: engine-specific language | High: engine-specific client library, some self-hostable | Low: plain queues and code | | Team discipline | Fit logic into states | Determinism rules | Reinvent retries and idempotency carefully | ## How a lead decides 1. **Inventory the jobs.** List multi-step jobs by duration, step count, volume, waits and fan-out. The mix decides more than any feature list. 2. **Count the re-solving.** How many teams maintain their own status tables, retry loops and stuck-job cleanup scripts? How many incidents in the last year were "a job silently stopped halfway"? 3. **Estimate the bill both ways.** Executions times steps, including retries and fan-out branches, under the engine's price shape, against current compute, storage and the engineering time spent on home-grown coordination. 4. **Pick a standard per class.** For example: long, waiting, branching or fan-out jobs go on the engine; two- or three-step chains at high volume stay as queue-linked functions. 5. **Decide the exit cost up front.** Keep business logic inside the step functions and the workflow thin, so a future move rewrites coordination, not the product. ## Making the standard stick - Publish a **golden path**: a template, a reference workflow, dashboards and alarms, and the criteria for each option. - Make **versioning rules** part of review: version gates, pinned function versions, replay tests. - Put **budget alarms** on engine charges per team, so a high-volume flow on the wrong option is noticed in weeks, not at year end. - Revisit the standard when the evidence changes: a new engine price shape, a provider move, or a workload class that outgrew its option.
- How would you estimate a hosted workflow engine's bill before standardising on it?Take each candidate workflow's monthly executions and multiply by steps per execution, counting retries, callback steps and every fan-out branch, then apply the engine's price shape. Compare that with what the same coordination costs today: queue and storage charges plus the engineering hours spent maintaining status tables and stuck-job tooling. Short, high-volume flows usually come out on the code side.
- What signals tell you that coordination in application code has outgrown itself?Teams each maintain their own status tables and retry loops; incidents read 'the job stopped halfway and nobody noticed'; answering 'where is this customer's order?' needs log archaeology across several functions; adding a wait or a fan-out takes weeks. When those show up in several teams at once, a shared engine pays for its standing costs.
- A hosted workflow engine is always cheaper because you stop paying for idle waiting.
- Put every multi-step flow on the engine, including high-volume sub-second chains.
- Coordination in application code is free because it reuses services we already run.
- Workflow definitions are portable, so switching providers later is a simple redeploy.
- Pick the engine with the longest feature list and make it mandatory for every team.