skip to content

Should a platform team standardise multi-step function jobs on a hosted workflow engine or keep coordination in application code, and what decides it?

level: principalimportance: should knowfreq 30%

answer

  1. what home-grown coordination keeps re-solving
  2. visibility of every single run
  3. price per step at volume
  4. the definition language as lock-in
  5. a standard per class of workflow

basics

~20 s

Standardise on a hosted engine for long, branching, waiting or fan-out workflows where per-execution history and built-in retries pay off; keep short, high-volume chains in code. Weigh per-step cost, definition-language lock-in and team discipline, not features alone.

solid answer

~50 s

It is a standard per **class of workflow**, not one answer for everything. A hosted engine earns its place where jobs run long, wait on people or callbacks, fan out, and need an **auditable per-execution history**: it replaces the status tables, retry loops and stuck-job scripts every team otherwise rebuilds. It adds standing costs: **per-step pricing** that punishes short, high-volume flows; a **definition language** or client library specific to that engine, which is lock-in; payload and history caps; and, for code-first engines, **determinism discipline** on every developer. Coordination in code (a queue between functions plus a status record) stays cheaper and simpler for two- or three-step chains at high volume. I would decide from evidence: incidents caused by lost or stuck jobs, how many teams maintain their own coordination, and a step-count times volume cost estimate, then publish a golden path with both options and the criteria that pick between them.

go deeper

for a junior

Recall the two options: a hosted engine that keeps workflow state and history, or functions linked by queues with state kept in your own database.

for a middle

Explain what an engine provides out of the box (durable state, retries, waits, history) and what coordination in code has to rebuild.

for a senior

Show the operating evidence for each option: stuck-job incidents, history-based debugging, per-step cost at volume, and payload and history caps.

for a principal

Own the standard: a per-class rule backed by a cost estimate and incident data, a golden path, versioning rules, and a planned exit cost.

## What the decision really is Every organisation that runs functions ends up with multi-step jobs: extract then review then publish, charge then provision then notify. The steps have to be coordinated somewhere. The choice a platform lead faces is whether that coordination becomes a **shared standard on a hosted workflow engine** or stays **inside each team's application code**, usually a queue between functions plus a status record in a database and some retry logic. This is not the choreography versus orchestration question. Both options here are orchestration; the question is who runs the orchestrator and who pays for it. ## What a hosted engine buys - **Durable state for free.** Executions survive crashes, deploys and days-long waits without each team designing a status table. - **Built-in retries, timeouts and callback waits**, configured instead of hand-written and re-tested by each team. - **Per-execution history.** An operator can open one document's run and see every step, input, output and error. In-code coordination usually reconstructs this from logs. - **Fan-out with a concurrency cap** and a fan-in, without building a job tracker. - **A shared operating model**: one set of dashboards, alarms and runbooks across teams. ## What it costs, every month - **Per-step pricing.** Many engines bill per state transition or per step. A long workflow with few steps is cheap; a short chain run millions of times can cost more in engine charges than in function compute. - **Lock-in.** A definition language or durable-execution client library is specific to its engine. Moving providers means rewriting workflows, not redeploying them. - **Hard caps.** Most engines cap the data passed between steps and, often, the size of an execution's history, which pushes designs toward storage references and batching. - **Discipline.** Code-first engines require deterministic workflow code; definition-based engines push complex logic into an awkward language. Either way, teams need training and review. - **Versioning work.** Long-running executions outlive deploys, so every structural change needs version gates or pinned function versions. - **A platform dependency.** The engine's outages and limits become everyone's outages and limits. ## Comparing the options | Concern | Definition-based hosted engine | Code-first durable engine | Coordination in application code | |---|---|---|---| | Long waits and callbacks | Built in | Built in | Status table plus a scheduler | | Run visibility | Execution history with a visual graph | Execution history and replay | Logs and status rows | | Cost shape | Per step, or duration on some types | Per action or self-run hosts | Function compute plus queue and storage | | Flexibility of logic | Limited to the definition language | A general-purpose language | A general-purpose language | | Lock-in | High: engine-specific language | High: engine-specific client library, some self-hostable | Low: plain queues and code | | Team discipline | Fit logic into states | Determinism rules | Reinvent retries and idempotency carefully | ## How a lead decides 1. **Inventory the jobs.** List multi-step jobs by duration, step count, volume, waits and fan-out. The mix decides more than any feature list. 2. **Count the re-solving.** How many teams maintain their own status tables, retry loops and stuck-job cleanup scripts? How many incidents in the last year were "a job silently stopped halfway"? 3. **Estimate the bill both ways.** Executions times steps, including retries and fan-out branches, under the engine's price shape, against current compute, storage and the engineering time spent on home-grown coordination. 4. **Pick a standard per class.** For example: long, waiting, branching or fan-out jobs go on the engine; two- or three-step chains at high volume stay as queue-linked functions. 5. **Decide the exit cost up front.** Keep business logic inside the step functions and the workflow thin, so a future move rewrites coordination, not the product. ## Making the standard stick - Publish a **golden path**: a template, a reference workflow, dashboards and alarms, and the criteria for each option. - Make **versioning rules** part of review: version gates, pinned function versions, replay tests. - Put **budget alarms** on engine charges per team, so a high-volume flow on the wrong option is noticed in weeks, not at year end. - Revisit the standard when the evidence changes: a new engine price shape, a provider move, or a workload class that outgrew its option.

  • How would you estimate a hosted workflow engine's bill before standardising on it?
    Take each candidate workflow's monthly executions and multiply by steps per execution, counting retries, callback steps and every fan-out branch, then apply the engine's price shape. Compare that with what the same coordination costs today: queue and storage charges plus the engineering hours spent maintaining status tables and stuck-job tooling. Short, high-volume flows usually come out on the code side.
  • What signals tell you that coordination in application code has outgrown itself?
    Teams each maintain their own status tables and retry loops; incidents read 'the job stopped halfway and nobody noticed'; answering 'where is this customer's order?' needs log archaeology across several functions; adding a wait or a fan-out takes weeks. When those show up in several teams at once, a shared engine pays for its standing costs.

  • A hosted workflow engine is always cheaper because you stop paying for idle waiting.
  • Put every multi-step flow on the engine, including high-volume sub-second chains.
  • Coordination in application code is free because it reuses services we already run.
  • Workflow definitions are portable, so switching providers later is a simple redeploy.
  • Pick the engine with the longest feature list and make it mandatory for every team.