XComs and Data Passing

## What an XCom is Airflow tasks do not share memory. Each task instance runs as its own process, often on a different worker machine or in a different Kubernetes pod, so a Python variable set in one task simply does not exist in the next. XCom ("cross-communication") is the mechanism Airflow provides to close that gap: a small, keyed value written by one task and read by another within the same DAG run. An XCom row is identified by the DAG id, the run, the task id, the key, and (for mapped tasks) the map index. That composite identity is why the same task pushing on Monday and Tuesday produces two separate values rather than overwriting one. ## Pushing a value There are two ways to push. **Explicitly**, from inside a Python callable, using the task instance object that Airflow passes into the context: ```python def extract(ti): ti.xcom_push(key="row_count", value=42) ``` **Implicitly**, by returning a value. Every operator inherits a `do_xcom_push` parameter that defaults to `True`; when the task finishes, its return value is stored under the reserved key `return_value`. A `PythonOperator` pushes whatever the callable returns; a `BashOperator` pushes the last line the command wrote to stdout. Setting `do_xcom_push=False` on the operator turns that off, which is what you do when an operator returns something big or uninteresting. ## Pulling a value ```python def load(ti): count = ti.xcom_pull(task_ids="extract", key="row_count") ``` If you leave `key` out, `xcom_pull` looks for `return_value` — the implicit push. If you pass a list of task ids, you get a list of values back, in the order of the ids you asked for. Templated fields can pull too, because the task instance is available in the Jinja context: `bash_command="echo {{ ti.xcom_pull(task_ids='extract') }}"`. By default a pull is scoped to the current DAG run. `include_prior_dates=True` widens the search to earlier runs, which is occasionally useful for "what was the last watermark" patterns but makes the DAG's behaviour depend on history — a trap for backfills. ## Pulling is not a dependency This is the single most common beginner error. `xcom_pull` is a database lookup executed at runtime; it tells the scheduler nothing. If you write a `load` task that pulls from `extract` but never wire `extract >> load`, the scheduler is free to run them in parallel, and `load` will very likely pull `None`. You must still declare the edge — either with the bitshift operators, or by using the TaskFlow API, where passing one task's output into another call creates the edge for you. ## What belongs in an XCom Because values are serialized into a row in Airflow's own metadata database, XCom is a control-plane channel, not a data-plane one. Good payloads are small facts: a row count, a partition date, a job id returned by an external system, an S3 or GCS URI naming where the real data was written. Bad payloads are the data itself — a DataFrame, a file's contents, a large JSON document. The accepted pattern is that the upstream task writes the dataset to object storage or a warehouse table and pushes only the pointer. XComs are also not the right place for credentials. Values are stored unencrypted and are rendered in the web UI, unlike Connections and Variables, which use Airflow's Fernet encryption. Secrets belong in a Connection or a secrets backend. ## When a pull returns None A `None` from `xcom_pull` almost always means one of: the `task_ids` or `key` string does not match what was pushed (they are plain strings, and typos fail silently); the upstream operator had `do_xcom_push=False`; the upstream callable simply returned `None`; the two tasks are not in the same DAG run; or there is no dependency, so the pull ran first. Debugging starts by opening the XCom tab for the upstream task instance in the UI and looking at exactly which keys exist. ## Related concepts you should not confuse with it Airflow Variables are deployment-wide key/value configuration read at parse or run time; Connections hold credentials and endpoints. Both are global and long-lived. An XCom is per-task, per-run, and transient — it is about *this* execution of *this* pipeline, and it disappears when the run's rows are cleaned out of the metadata database.