Why does a data pipeline task need an execution timeout as well as a retry policy?

## Two different failure shapes Orchestration has to survive two things that look nothing alike. A task can **die** — non-zero exit, uncaught exception, worker lost — and a task can **hang**, sitting in a state where it makes no progress and produces no error. Retries handle the first. Nothing in a retry policy handles the second, because retries are triggered *by a failure*, and a hang never produces one. From the scheduler's point of view a hung task is indistinguishable from a healthy long-running one: the process is alive, the heartbeat is fine, the state is "running". Left alone it can stay that way for hours or days. ## What a hang actually costs - **A worker slot is occupied.** In a bounded pool, one wedged task permanently removes capacity; a handful of them starve every other pipeline on the platform. - **Downstream stays queued.** Dependants are waiting on a task that will never resolve, so the whole branch stalls without ever entering a failed state. - **The next run collides.** A daily pipeline whose 02:00 run is still "running" at 02:00 the next day either runs concurrently with itself — two writers over the same target — or queues, depending on the concurrency policy. Either way the backlog grows one run per period. - **Failure alerting stays silent.** Alerts keyed on the failed state never fire, because nothing failed. This is the classic "we found out from the business at 10am" incident. ## Where hangs come from Almost always an unbounded wait at an I/O boundary: an HTTP or database client with no socket read timeout, a query blocked behind a lock or a queue with no statement timeout, a poll loop with no deadline, a distributed job whose driver is alive while an executor is gone, a `fetch`-style call against a source that accepted the connection and then went quiet. The deep fix is a timeout on the inner call. The orchestrator's execution timeout is the outer safety net for the ones you did not anticipate. ## The layers of timeout Different orchestrators name these differently, but conceptually there are three: 1. **Task execution timeout** — maximum wall time for a single attempt. Kills the attempt and marks it failed. 2. **Run (pipeline) timeout** — maximum wall time for the whole run, end to end. This catches the case where no individual task is pathological but the run as a whole has drifted far past its window — fifty tasks each 30% slower than usual. 3. **Deadline / SLA monitor** — not a kill switch at all, but a clock: "this must be done by 06:00". It fires a notification while the run continues, and it is the one that maps to a promise made to a consumer. The first two protect the *platform*; the third protects the *consumer*. Teams that configure only the first two learn about lateness from a dashboard user. ## Sizing a timeout Use the distribution, not a guess. Take the task's observed high-percentile duration over a representative period and add headroom for legitimate growth — month-end volumes, a backfill-sized partition, a slower shared warehouse. A timeout set at the median kills healthy runs and manufactures incidents; a timeout of 24 hours on a 5-minute task technically exists but never protects anything. Also revisit it as data grows: yesterday's generous ceiling becomes today's flaky failure when volume doubles. ## The arithmetic everyone gets wrong The execution timeout applies **per attempt**, not to the task overall. A task with a 2-hour timeout and 2 retries can consume roughly 3 × 2 hours, plus two retry delays, before it reports a terminal failure — six hours of a pipeline that was supposed to be done in twenty minutes: ```text attempt 1: 02:00 -> 04:00 killed by timeout wait 15m attempt 2: 04:15 -> 06:15 killed by timeout wait 15m attempt 3: 06:30 -> 08:30 killed by timeout task failed at 08:30; alert fires; deadline was 06:00 ``` If the promise is 06:00, this configuration cannot keep it even when the safety nets work exactly as designed. Timeout, retry count and retry delay are one budget, and it has to be sized against the deadline as a whole. A hung task on a tight deadline usually wants a shorter timeout and fewer attempts, so the human is woken while there is still time to act. ## Choosing what the timeout does Killing is not always right. For a task that is genuinely just slow — a large but healthy load — killing at the ceiling and retrying may be strictly worse than letting it finish while notifying someone. That is the argument for pairing a *hard* timeout well above any plausible healthy duration with a *soft* deadline notification at the level you actually care about: the notification buys a human decision, and the hard timeout stops the pathological case from consuming the platform forever.