Where does Airflow store XCom values by default, and what limits their size?

## Where the value actually lands Out of the box, XCom has no separate storage system. A push becomes an `INSERT` into the `xcom` table of the Airflow metadata database — the same PostgreSQL or MySQL instance that holds DAG runs, task instances, logs metadata and the scheduler's own bookkeeping. The row carries the dag id, the run, the task id, the key, the map index and the serialized value. That single design fact explains nearly everything about how XCom behaves and why interviewers keep asking about size. ## Serialization Airflow serializes XCom values to JSON. Practically, that means dicts, lists, strings, numbers and booleans go through cleanly, and anything else — a pandas DataFrame, a database connection, a custom class, a `set` — raises a serialization error when the task finishes rather than being silently stored. Airflow adds handling for a few extra types it knows about, but the mental model to interview with is "if it isn't JSON, it doesn't fit." Older Airflow releases had a pickling switch; do not reach for it, because unpickling attacker-controlled bytes in the scheduler and workers is a security problem, not a size solution. ## The hard ceiling comes from the column The value column is a binary column, and the maximum size is whatever that column type allows on your database engine. The Airflow documentation calls this out explicitly, and the numbers differ per backend: MySQL's BLOB tops out around 64 KB, PostgreSQL's BYTEA around 1 GB, SQLite's BLOB around 2 GB. The consequence is that the *same DAG* can work on one deployment and fail on another. A team that developed against PostgreSQL and deployed onto a MySQL-backed managed Airflow will find tasks failing on the write with a data-too-long error, and the DAG code did not change at all. ## The soft ceiling matters more Even far below the hard limit, large XComs are a bad idea, because the metadata database is a shared, latency-sensitive resource: - **Scheduler contention.** The scheduler queries this database constantly to decide what to queue. Rows measured in megabytes make the table bigger, the indexes colder and the vacuum/maintenance load heavier. - **Every read is a round trip.** A downstream task pulling a 50 MB blob pulls it over the wire from the database, not from a nearby cache. - **The UI renders XComs.** The grid view's XCom tab reads the value; huge values make pages crawl. - **Backups and restores.** Your metadata database backup is now dominated by pipeline payloads that were never meant to be durable. A good rule of thumb to state in an interview: keep XComs in the kilobyte range, treat anything over a megabyte as a design smell, and never let a payload's size scale with the size of the data being processed. ## Lifecycle: nothing expires on its own XCom rows are not TTL'd. They persist for as long as their DAG run's records do. Two mechanisms remove them: - Clearing a task instance deletes the XComs that task instance produced, so a rerun starts clean rather than reading a stale value. - `airflow db clean --clean-before-timestamp <ts>` trims old rows from history tables, including `xcom`. On a busy deployment this is a scheduled maintenance job, not a one-off. A subtle operational consequence: if `db clean` has already removed an old run's XComs and you then clear and rerun a *downstream-only* task in that run, its pull finds nothing. That is one more argument for downstream tasks deriving what they need from the run's logical date rather than depending on a pushed value surviving. ```bash airflow db clean --clean-before-timestamp 2024-01-01 --yes ``` ## The escape hatch If you genuinely want bigger payloads to flow through the XCom API, you change *where* they are stored rather than raising the limit: configure a custom XCom backend, a class that serializes the value into object storage and puts only a small pointer in the database row. That keeps `ti.xcom_pull` looking the same to DAG authors while the bytes live in S3 or GCS. It is a deployment-wide decision, and it brings its own lifecycle problem — deleting the database row does not delete the object. ## What to say when asked "how big can an XCom be?" The strong answer is not a number. It is: "the hard limit depends on the metadata database column type, but I design as if the budget is a few kilobytes, because XCom is a control channel and its cost lands on the scheduler's database."