How do you get alerted when an Airflow task fails, and what do the callback arguments give you?

## The two mechanisms Airflow ships with a plain email path and a general callback path. **Email** is controlled by three `BaseOperator` arguments: `email` (address or list), `email_on_failure` and `email_on_retry`. It requires a working SMTP configuration in `airflow.cfg` or the corresponding environment variables. It is the oldest mechanism, it only sends mail, and it has no routing, deduplication or escalation. It is fine for a hobby deployment and rarely sufficient for a production one. **Callbacks** are arbitrary Python callables invoked at defined lifecycle points: - `on_execute_callback` — just before the task's execute begins. - `on_success_callback` — after a successful run. - `on_retry_callback` — on each retry, i.e. every time the instance enters `up_for_retry`. - `on_failure_callback` — when the instance reaches terminal `failed`, after retries are exhausted. The same `on_failure_callback` and `on_success_callback` names also exist on the DAG object, where they fire for the **DAG run** rather than an individual task — useful for a single 'pipeline X failed' notification instead of one message per failed task. ## The context dictionary Every callback receives one argument, conventionally called `context`: a dict of the same values Jinja templates see. The useful keys for alerting are `dag`, `task`, `ti` (the task instance, from which you get `try_number`, `state` and `log_url`), `run_id`, `logical_date`, `data_interval_start` / `data_interval_end`, and `exception` — the error object that caused the failure. A good alert names the DAG and task, the run's interval, the attempt number, a one-line error summary, and a deep link built from `ti.log_url` so the responder lands directly on the failing log. ```python def notify_failure(context): ti = context["ti"] send_slack( f"FAILED {ti.dag_id}.{ti.task_id} " f"run={context['run_id']} try={ti.try_number}\n" f"{context.get('exception')}\n{ti.log_url}" ) ``` Wiring it once in `default_args` is the right default so nobody ships a silently-failing task: ```python default_args = {"on_failure_callback": notify_failure, "retries": 2} ``` For Slack specifically, the community provider ships helpers (a Slack webhook notifier and the Slack operators) so you rarely have to hand-roll the HTTP call, but the shape is the same: a callable that closes over a connection id and formats the context. ## Timing: retries delay the page The most important operational detail is that `on_failure_callback` fires only when the task reaches terminal `failed` — that is, after every retry has been exhausted. A task with `retries=4` and `retry_delay=timedelta(minutes=10)` will not alert for something like forty minutes after the first symptom. If that is too slow for a pipeline with a downstream commitment, you have three levers: reduce retries for that task, emit a lower-severity signal from `on_retry_callback` so you can *see* the flapping before it pages, or attach the real deadline to an SLA rather than to failure. Using `on_retry_callback` as a metric rather than a page is a good pattern generally: increment a counter or post to a low-noise channel, so a task quietly succeeding on its third attempt every night becomes visible instead of invisible. ## Where callbacks run, and why it matters A task-level callback executes in the task's own process on the worker, as part of finishing the instance. Three consequences follow. First, the callback must be **fast** — a slow HTTP call to a paging provider holds a worker slot. Second, it must not **throw**: an exception inside a failure callback is logged but leaves you with a failed task and no notification, the worst combination, so wrap the delivery in a try/except. Third, it needs whatever credentials it uses to be reachable from the worker, normally via an Airflow connection rather than an inline secret. A callback also cannot fire if the process never got to run it — the zombie case, where the worker was OOM-killed or the pod evicted. There the scheduler fails the instance and the failure callback does get invoked from that path, but any assumption that your notifier always runs inside the dying process is wrong. ## What good alerting looks like Beyond the mechanics, interviewers want the judgment layer: - **Route by severity, not by task.** Everything paging is the same as nothing paging. A failed reporting refresh belongs in a channel; a failed regulatory extract pages. - **Alert on the pipeline, not each task.** A DAG-level `on_failure_callback` avoids twenty messages when a fan-out of twenty mapped tasks all fail on the same upstream outage. - **Alert on staleness too.** A DAG that never started — because it was paused, or the scheduler was down — produces no failure and therefore no callback. Failure alerting is blind to absence; freshness checks on the output table or an SLA are what catch it. - **Include the deep link.** The single biggest quality difference between alerts is whether the responder can reach the log in one click. - **Emit metrics as well as messages.** Airflow can emit StatsD metrics and, in recent versions, OpenTelemetry, giving you dashboards for task duration and failure counts that no per-event notification provides.