In Airflow, why does top-level code in a DAG file slow down the whole scheduler?

## Two clocks: parse time and run time A DAG file is a Python program whose job is to *describe* a graph. It is executed by the parsing component (the scheduler's DAG file processor in Airflow 2; a mandatory standalone DAG processor in Airflow 3) again and again, whether or not any DAG run is happening — and often by the webserver's and workers' own imports as well. Run time is different. Task bodies execute exactly once per task instance, on a worker, when the scheduler decides that instance is eligible. Almost every Airflow performance question reduces to knowing which of these two clocks a given line of code is on. ## The parsing loop The processor scans the DAGs folder for files (`dag_dir_list_interval` controls how often it looks for new ones), imports each one, collects the `DAG` objects in its globals, checks for cycles, and writes a JSON serialization into the metadata database. `min_file_process_interval` sets the minimum gap before the same file is parsed again — short enough that the loop feels continuous. The serialized form is what the scheduler and the UI read afterwards, which is why the UI can render a DAG without importing your code. So the arithmetic is brutal: a two-second top-level HTTP call, in one file, parsed continuously, is two seconds of processor time burnt repeatedly, forever. Multiply by a hundred files with the same habit and DAGs across the whole deployment start being scheduled late, with no single DAG obviously at fault. There is also an import timeout (`dagbag_import_timeout`); a file that exceeds it is abandoned and its DAGs vanish from the UI until a parse succeeds — an outage whose cause is invisible unless you look at import errors. ## What belongs where ```python # BAD - executes on every parse of this file import requests tables = requests.get("https://config.internal/tables").json() with DAG("ingest", schedule="@daily", start_date=START, catchup=False): for t in tables: PythonOperator(task_id=f"load_{t}", python_callable=load, op_args=[t]) ``` ```python # GOOD - the call happens on a worker, once per run @task def fetch_tables() -> list[str]: return requests.get("https://config.internal/tables").json() load.expand(table=fetch_tables()) ``` The general rule: at module level, only cheap, deterministic, local expressions — constants, small literal lists, a config file already on disk, and the DAG and operator constructors themselves. Everything that touches a network, a database, a secrets backend or a large library goes inside a task body or its callable, not at import. Two refinements. First, **imports count**: `import pandas` at the top of a DAG file is paid on every parse of that file, so move heavy imports inside the callable. Second, **connections and Variables**: `Variable.get("x")` at module level hits the metadata database on every parse; fetch it inside the task, or reference it through a Jinja template such as `{{ var.value.x }}`, which is rendered at run time rather than at parse time. ## Determinism matters as much as cost Top-level code that changes between parses gives you a DAG whose structure is unstable. `start_date=datetime.now()` is the canonical example: every parse moves the start date, so the scheduler's idea of which intervals exist keeps shifting. A task list derived from a live API means tasks silently appearing and disappearing between parses, leaving orphaned task instances from earlier runs. Pin the structure to something stable — a fixed timestamp, a config file committed to the repo — and push volatility into run time. ## Diagnosing it Airflow gives you direct measurements. `airflow dags report` shows per-file parse duration and how many DAGs each produced; the scheduler exposes DAG-processing metrics; and the import-errors view lists files that timed out or raised. Sort by parse duration and the offenders are usually obvious — one file taking seconds while the rest take milliseconds. Mitigations, in order: remove the work; if it genuinely must be parse-time, cache it into a file the parse reads cheaply; raise `min_file_process_interval` to reduce frequency; and split enormous DAG files so a slow one does not delay the rest. ## The interview answer Say the two-clocks framing first, then the loop, then the concrete consequences — scheduler lag across unrelated DAGs, import timeouts making DAGs disappear, structural instability — and finish with the fix. That progression shows you have operated Airflow rather than only authored DAGs on it.