In Prefect, why might an on_failure hook on a flow never fire in production?

## What a hook is Prefect lets you attach callables to state transitions on a `@flow` or `@task`: ```python def alert(flow, flow_run, state): post_to_slack(f"{flow_run.name} -> {state.name}: {state.message}") @flow(on_failure=[alert], on_crashed=[alert], on_cancellation=[alert]) def pipeline(): ... ``` Flow hooks receive `(flow, flow_run, state)`; task hooks receive `(task, task_run, state)`. The available hooks mirror the state model — completion, failure, crash, cancellation, and running — and each takes a **list** of callables, so several can be registered. ## Reason one: the hook is bound to a state type `on_failure` fires for Failed-type states. That includes `TimedOut`, which is a Failed state under a more specific name — good news, since a hung task that blows its `timeout_seconds` does reach your alert. It does **not** include: - **Crashed** — the environment ended the run (OOM kill, pod eviction, spot reclaim, infrastructure that never started). Needs `on_crashed`. - **Cancelled** — someone stopped the run deliberately. Needs `on_cancellation`. This is the single most common production surprise: alerting is wired to `on_failure`, memory pressure starts killing containers, the runs go Crashed, and the on-call channel stays silent while the pipeline is down. ## Reason two: hooks run in the process that is dying Hooks are client-side. Prefect executes them in the same process that owns the run, after the state transition. That means: - A `SIGKILL` (OOM killer, `kill -9`, an evicted pod given no grace) leaves nothing to execute the callback. Even `on_crashed` cannot run locally if the machine is gone; in that case the Crashed state is recorded from outside, and no local code ran. - A graceful `SIGTERM` with enough shutdown time can let cancellation or crash handling run — which is why grace periods and handling shutdown properly are worth configuring. - A worker that loses network connectivity may execute the hook but fail to report anything. So the reliability of a hook is bounded by the reliability of the very process whose death you are trying to be told about. That is a circular dependency, and it is why crash-proof alerting cannot live only in hooks. ## Reason three: hooks fail quietly An exception inside a hook is logged and swallowed: it does not fail the run, does not retry, and does not change the state. That is deliberate — you do not want a flaky Slack webhook turning a successful pipeline into a failed one — but it means a hook that has been broken for a month looks exactly like a hook that never had anything to report. Hooks that matter deserve their own error handling, a timeout on any network call they make, and an occasional deliberate test. Hooks are also not retried and are not guaranteed to run exactly once in every pathological case, so treat them as best-effort notification and enrichment, never as the step that maintains correctness. Never put "release the lock", "commit the transaction" or "write the watermark" in a hook; put it in the task body where failure is visible and retried. ## What to do instead for alerting Prefect's server side emits **events** on state changes, and **automations** react to them: notify a channel, call a webhook, cancel or re-run something. Because that logic lives in the API rather than in the run's process, it still triggers when the process is gone — which is exactly the case you most need alerting for. The healthy split is: - **Automations** — "this deployment's run failed or crashed", "a run has been Running longer than N minutes", "no run completed in the last day". This is your paging path. - **Hooks** — in-process work that needs the run's own context: publishing a markdown artifact summarising what the failure found, closing a client, emitting a metric with local variables, writing a debug bundle. In Prefect 3, flow-run heartbeats give automations a way to notice a run that stopped reporting and mark it Crashed, closing the "stuck in Running forever" gap that hooks structurally cannot cover. ## Reviewing a hook setup Ask: which state types are actually covered? Does anything paging depend on the dying process? Would a hard kill of the container be noticed at all, and how fast? Is any correctness-critical step hiding in a hook? Does the hook itself have a timeout and a failure path? Most incidents in this area are not exotic — they are one hook, registered on one state type, in a system where the interesting failures happen in a different one.