Monitoring and Retries

## What the parameters mean Every Airflow operator inherits a set of retry-related arguments from `BaseOperator`. The two you set most often are: - **`retries`** — an integer count of attempts *after* the first one. `retries=0` means the task gets exactly one shot. `retries=3` means up to four executions of that task instance. - **`retry_delay`** — a `datetime.timedelta` saying how long to wait between a failure and the next attempt. Both are ordinary keyword arguments on any operator, and both are commonly hoisted into `default_args` so every task in the DAG inherits them. ```python from datetime import timedelta from airflow.decorators import dag, task default_args = { "retries": 3, "retry_delay": timedelta(minutes=5), } ``` ## What actually happens on a failure When a task instance raises, the worker records the failure and the scheduler decides what state to put it in. If the instance's try number is still below `retries + 1`, it becomes **`up_for_retry`** and gets a `next_retry_datetime` computed from `retry_delay`. The DAG run remains `running` — a task in `up_for_retry` does not fail the run. When the delay elapses, the scheduler queues the instance again and the try number increments. Only after the final attempt fails does the instance become **`failed`**, and only then do downstream tasks react (typically going to `upstream_failed` under the default `all_success` trigger rule). This is why a retried task's log page shows several numbered attempts: each try writes its own log file, and the UI lets you page between them. Reading only the last try is a common way to miss the actual cause, because an early attempt may show a different error than the final one. ## Backoff A fixed `retry_delay` retries at a constant cadence. That is fine for a flaky network call, but it hammers a downstream service that is genuinely down. Airflow offers `retry_exponential_backoff=True`, which grows the wait between successive attempts rather than keeping it flat, and `max_retry_delay` (also a `timedelta`) to put a ceiling on how long the gap can get. The combination gives you fast recovery from a one-off blip without a task pinned in `up_for_retry` for hours. ```python @task( retries=5, retry_delay=timedelta(seconds=30), retry_exponential_backoff=True, max_retry_delay=timedelta(minutes=20), ) def call_flaky_api(): ... ``` ## What retries do *not* fix Retries only help when the failure is transient — a dropped connection, a throttled API, a node evicted mid-run. They do nothing for a deterministic failure such as a syntax error, a schema mismatch, or a missing permission; those simply burn every attempt and delay the alert by `retries × retry_delay`. That delay matters: a task with five retries five minutes apart takes almost half an hour to reach `failed` and fire whatever alerting hangs off failure. Sizing retries is therefore also a decision about how quickly you want to be told. The second thing retries do not fix is non-idempotent work. Airflow re-executes the whole task body on each attempt; it does not resume from where the code stopped. If your task appended rows, sent emails, or incremented a counter before it failed, the retry does it again. Making the task safe to re-run — deleting-then-inserting for the run's data interval, using a `MERGE`, writing to a run-scoped path, or keying an external call on the interval — is a precondition for turning retries on at all, not an optimisation. ## Related knobs worth knowing - **`execution_timeout`** — a `timedelta` after which the task instance is killed and treated as failed, which then feeds the retry machinery. Without it, a hung task can occupy a worker slot indefinitely and no retry ever happens, because the attempt never ends. - **`on_retry_callback`** — a callable fired on each retry, useful for counting retries as a metric rather than only alerting on the terminal failure. - **`depends_on_past`** and `wait_for_downstream` interact with failures at the DAG-run level: a permanently failed instance can block later runs of the same task. ## Where to set them Put a conservative default in `default_args` — a couple of retries with a few minutes' delay covers most infrastructure flakiness — and override per task where the failure profile differs. A task that calls a rate-limited third-party API wants more attempts with exponential backoff; a task that runs an expensive warehouse query you do not want executed twice wants very few, or zero until the query itself is made idempotent.