How do you decide how much work belongs in a single orchestrated task?

## A task is four units at once Granularity questions get answered badly when people reason about "logical steps". Reason instead about the four roles a task actually plays: 1. **Unit of retry.** Retrying re-executes the whole task. Everything inside it happens again. 2. **Unit of observability.** Duration, state, logs and lineage are recorded per task. Anything inside is invisible to the platform. 3. **Unit of parallelism.** Tasks are what the orchestrator can run concurrently and place on workers. 4. **Unit of rerun blast radius.** Reruns operate on a task and its downstream closure. A task is the smallest thing you can redo. Every sizing rule below falls out of one of those. ## Split here **Different failure causes.** An API extract fails on rate limits and timeouts; a validation fails on bad data; a warehouse load fails on locks and permissions. Different causes deserve different retry counts and often different owners — and separate tasks tell you which one broke without reading a log. **A non-repeatable side effect.** Sending an email, calling a payment API, posting to a webhook, incrementing an external counter. Isolate these into their own task so a retry of the neighbouring compute cannot re-fire them, and so you can give them a retry policy of zero if that is what safety requires. The classic incident is a forty-minute task that emails a report at minute thirty-eight and fails at minute thirty-nine; three retries later, four copies have gone out. **Expensive work you do not want to repeat.** If a thirty-minute extract is followed by a two-minute load that fails intermittently, one task means every load failure costs thirty-two minutes. Splitting makes the retry cost two. **Meaningfully different resource shapes.** A memory-hungry transform and a trivial metadata update should not share sizing, a worker class, or a concurrency limit. ## Keep together here **Shared expensive local state.** If step two needs the in-memory or on-local-disk output of step one, splitting them forces you to materialise the intermediate to shared storage. Sometimes that is worth it; often the intermediate is worthless on its own and the write is pure cost. **One atomic effect.** The orchestrator gives you no transaction across tasks: if the second half fails, nothing undoes the first half. Work that must be all-or-nothing to consumers should either sit in one task that commits once, or be structured as write-to-staging tasks plus a final publish task that swaps atomically. **Trivial work.** A task that runs for two hundred milliseconds but takes seconds to schedule, queue and record is nearly all overhead. ## The two failure modes **Too coarse** looks like: one task called `run_pipeline`, all failures reported identically, a rerun redoing everything, no idea where time goes, and partial writes left behind by a mid-task failure that nothing cleans up. **Too fine** looks like: thousands of nodes, scheduler and metadata pressure, per-task startup dominating actual work, a graph nobody can read on screen, and logs fragmented across hundreds of task views so tracing one record means opening dozens of them. ## Do not rebuild an engine's parallelism in the graph A common mistake is generating one task per file for ten thousand files. The orchestrator becomes the bottleneck doing work a data engine does natively: hand it the prefix and let it parallelise internally. Use orchestrator-level fan-out when the units are genuinely independent, coarse (minutes each), individually retryable, and few enough to read — dozens to low hundreds, not tens of thousands. ## Atomicity, concretely A well-shaped task leaves exactly one durable effect that is visible either fully or not at all. Techniques: - write to a staging path or table, then swap, rename or repoint in the final step; - overwrite a whole partition rather than appending rows, so a rerun replaces rather than duplicates; - key writes so that a repeat is a no-op or an upsert. The test to apply to any task you design: *if this is killed halfway and retried, is the outcome the same as if it had run once cleanly?* If the answer is no, either fix the task or move the offending part out of it. ## A worked reshape A single forty-minute `daily_sales` task becomes: `extract_orders` (retry 3, network-flaky), `validate_orders` (retry 0 — bad data will not fix itself), `load_staging` (retry 2), `publish_swap` (retry 1, atomic), `notify` (retry 0, isolated side effect). Now a bad-data day fails in two minutes at validation with an obvious cause, lock contention on load retries cheaply, and no retry can double-send the notification. ## Granularity is also a communication choice Task names are what an on-call engineer reads at 3 a.m. and what appears in lineage and dashboards. A graph whose node names describe meaningful stages — extract, validate, load, publish — diagnoses itself; a graph of `step_1` through `step_12`, or one giant node, does not. If a boundary makes the failure message obviously actionable, that is evidence it is the right boundary. ## Interview framing Lead with the four units, give the split-here and keep-together rules, name both failure modes, and finish with the retry test. Mentioning the isolated side effect and the atomic publish is what marks the answer as coming from operating pipelines rather than reading about them.