In Airflow, when does a DAG scheduled on Datasets actually run?

## The idea Time-based scheduling encodes a *guess*: "upstream is usually done by 03:00, so start at 03:30". Data-driven scheduling encodes the *actual* dependency: "start when the orders table has been refreshed". Airflow 2.4 introduced Datasets for this; Airflow 3 renames them to Assets and extends them with external watchers. ## The mechanics A `Dataset` is constructed from a URI string: `Dataset("s3://warehouse/orders")`. The producer declares it as an outlet on a task; the consumer lists it in `schedule`. ```python from airflow.datasets import Dataset orders = Dataset("s3://warehouse/orders") # producer with DAG("ingest_orders", schedule="@daily", start_date=..., catchup=False): PythonOperator(task_id="write_orders", python_callable=write, outlets=[orders]) # consumer — no schedule of its own with DAG("orders_mart", schedule=[orders], start_date=..., catchup=False): PythonOperator(task_id="build_mart", python_callable=build) ``` When `write_orders` reaches the `success` state, Airflow writes a **dataset event** against that URI. The scheduler watches consumers: `orders_mart` becomes eligible once `orders` has at least one event newer than the consumer's last run. With several datasets in the list, the default is conjunctive — *all* of them must have been updated since the last run, so a consumer of a daily and a weekly feed runs weekly, not daily. Airflow 2.9 added `DatasetAny(...)` and `DatasetAll(...)` for explicit OR/AND expressions and a combined time-or-dataset schedule for "run when data arrives, but at least once a day regardless". ## What the URI is and is not The URI is a **name**, not a location Airflow polls. Nothing verifies that anything was written to `s3://warehouse/orders`. The consequences follow directly: - A producing task that succeeds without writing a byte still emits the event, and the consumer runs on stale data. - A task that writes the data but then fails on a later line emits **no** event, and the consumer silently never runs — a stall with no error anywhere, which is the failure mode teams find hardest to notice. - Producer and consumer must spell the URI identically. A trailing slash or a differing scheme creates two unrelated datasets and a dependency that quietly does not exist. Because the semantics are "a task succeeded", the granularity of the guarantee is the granularity of the task. Make the outlet-bearing task the one whose success genuinely means "the data is complete and readable". ## Scheduling implications **No data interval.** A dataset-triggered run is not derived from a timetable, so there is no window it represents. Do not build `WHERE created_at >= data_interval_start` logic in a dataset-scheduled DAG; pass the window explicitly (through the triggering run's own knowledge of it, or by having the consumer read a watermark) or design the consumer as a full refresh of whatever is currently present. **No catchup.** There is nothing to catch up: dataset events are point-in-time facts, not a sequence of intervals. If you need to replay history through a dataset-scheduled DAG, you trigger it manually or backfill it as a DAG with an explicit range — the dataset mechanism will not reconstruct a past sequence for you. **Event coalescing.** If a dataset is updated several times while the consumer is already running, the consumer does not run once per event; it becomes eligible again after it finishes. Consumers should therefore be written to process "everything new", not "exactly one upstream batch". **Deployment scope.** Datasets link DAGs inside one Airflow deployment. Two separate Airflow instances do not share dataset events by that mechanism alone. ## When to reach for it Datasets are the right tool when the real dependency is data readiness and the producers' timing is variable — an upstream job that sometimes finishes at 03:10 and sometimes at 06:40. They remove the brittle offset guessing that a cron-plus-sensor arrangement encodes, they show the dependency in the UI's dataset graph, and they let a producer's schedule change without every consumer needing an edit. They are the wrong tool when the consumer needs a defined data window per run (an interval-partitioned incremental load), when the dependency crosses Airflow deployments, or when you need historical replay through the same path, since dataset triggering has no notion of an interval to replay. ## Operational advice Monitor for **absence**: a dataset-scheduled DAG that has not run in longer than its expected cadence is the signal that an upstream task failed before emitting its event. Because nothing is scheduled, there is no failed run to alert on — the DAG simply goes quiet. That is the single most important alert to add when a team adopts data-driven scheduling.