An Airflow ExternalTaskSensor in a 06:00 DAG times out waiting on a task in a 02:00 DAG. Why?

## What the sensor actually queries `ExternalTaskSensor` is cross-DAG dependency by lookup, not by subscription. On each poke it asks the metadata database, in effect: *does a run of `external_dag_id` exist with logical date X, and is `external_task_id` in that run in one of `allowed_states`?* The key detail is where X comes from. By default X is **the sensor's own run's logical date** — not "the most recent run", not "today's run". That default is deliberate and correct for the intended case: two DAGs on the *same* schedule, where matching intervals really do correspond. It is wrong for everything else, and the failure is silent: no error, no missing-object exception, just a sensor that pokes patiently until it times out, every single day. ## Making the calendars line up - **`execution_delta`** — a `timedelta` *subtracted* from the sensor's logical date to produce the one it looks for. A 06:00 DAG waiting on that morning's 02:00 run uses `execution_delta=timedelta(hours=4)`. Getting the sign backwards is the most common follow-up bug, and it presents identically: perpetual timeout. - **The callable form** — a function mapping your logical date to the target's, for cases a fixed delta cannot express: a daily DAG waiting on the last hourly run, a weekday DAG waiting on Friday's run, a monthly roll-up waiting on the month's final daily run. Return a single date or a list of dates when you must wait for several runs. - You cannot pass both a delta and a callable — pick one. ## Hardening arguments people forget - **`check_existence=True`** makes the sensor verify that the referenced DAG and task actually exist and fail immediately if not. Without it, a renamed task or a typo in `external_dag_id` produces the same symptom as everything else — a long silent wait. - **`failed_states`** lets the sensor treat upstream failure as failure. Left unset, an upstream run that failed at 02:05 leaves the sensor poking until its own timeout hours later, delaying the alert and burning capacity for a result that will never come. - **`allowed_states`** defaults to success but can include others; and `external_task_id=None` waits on the whole DAG run rather than a single task. - **`mode='reschedule'` or `deferrable=True`** — this is a long wait by construction, so it should not hold a worker slot in poke mode. ```python ExternalTaskSensor( task_id="wait_for_ingest", external_dag_id="ingest_events", external_task_id="publish", execution_delta=timedelta(hours=4), failed_states=["failed", "skipped"], check_existence=True, mode="reschedule", poke_interval=300, timeout=60 * 60 * 3, ) ``` ## Backfills are where this really bites Because the match is on logical date, a backfill of the downstream DAG looks for upstream runs at the corresponding historical dates. If the upstream DAG was created later, was renamed, or has had its history cleaned up, those runs do not exist and every backfilled sensor times out — hours of poking per run before anything reports a problem. Anyone who has backfilled a month of a cross-DAG pipeline has felt this; saying so is what marks the answer as lived rather than read. ## The design point Even configured perfectly, this sensor couples two DAGs through a polling loop over the metadata database and inherits the fragility of a shared calendar. The alternatives worth naming: have the upstream DAG trigger the downstream one explicitly at the end of its work; merge the two DAGs if they are genuinely one pipeline with one cadence; or move to data/asset-driven scheduling so the downstream is started by the upstream's output rather than by a clock plus a poll. A senior answer fixes the timeout *and* asks whether the dependency should have been expressed by polling at all.