How do you configure a custom XCom backend in Airflow, and what must the class implement?

## Why a custom backend exists By default, an XCom value is serialized and stored as a row in Airflow's metadata database, which caps the useful payload at kilobytes and puts any bloat on the scheduler's own database. A custom XCom backend changes *where* the bytes live without changing the API DAG authors use: `ti.xcom_push`, `ti.xcom_pull` and TaskFlow argument passing all look identical, while the payload is transparently offloaded to object storage and only a pointer is written to the database. ## Configuring it One setting, deployment-wide: ```ini [core] xcom_backend = my_company.airflow.xcom.S3XComBackend ``` or equivalently `AIRFLOW__CORE__XCOM_BACKEND=my_company.airflow.xcom.S3XComBackend`. There is no per-DAG or per-task override; the choice applies to every XCom in the installation, which is the first thing to say when someone proposes one "just for our team's DAG." ## The class contract The class subclasses `airflow.models.xcom.BaseXCom` and implements the serialize/deserialize pair: ```python from airflow.models.xcom import BaseXCom class S3XComBackend(BaseXCom): PREFIX = "s3xcom://" BUCKET = "company-airflow-xcom" @staticmethod def serialize_value(value, **kwargs): key = build_key(**kwargs) # from dag_id / task_id / run_id / key write_object(S3XComBackend.BUCKET, key, value) reference = f"{S3XComBackend.PREFIX}{S3XComBackend.BUCKET}/{key}" return BaseXCom.serialize_value(reference) @staticmethod def deserialize_value(result): reference = BaseXCom.deserialize_value(result) if isinstance(reference, str) and reference.startswith(S3XComBackend.PREFIX): return read_object(reference) return reference ``` The shape is always the same: `serialize_value` puts the payload somewhere durable and returns something small enough for the column; `deserialize_value` recognises its own pointers and resolves them, passing anything else straight through so existing small XComs still work. The exact signatures of these methods have changed across Airflow releases — check them against the version you are running rather than copying a blog post, because a mismatched signature fails at runtime on every task, not at import. ## `orm_deserialize_value`: the one people miss `BaseXCom` also has `orm_deserialize_value`, used when Airflow needs a representation of the value for display and bookkeeping rather than for a task to consume — most visibly, rendering the XCom tab in the UI. If you do not override it, the webserver will happily download every offloaded object just to draw a table. Overriding it to return the pointer string (or a short summary) keeps the UI fast and keeps webserver processes out of your data path. ## Deployment is where this actually goes wrong The class is imported by the scheduler, every worker, the triggerer and the webserver. If it lives in a package installed only in the worker image, tasks will run fine and the UI will raise import errors when it tries to render XComs — a confusing failure that shows up days later. Ship the backend in a package installed into the same base image every component uses, and make sure the credentials or IAM role for the storage bucket are available to all of them too. Putting the class in the DAGs folder is tempting and fragile: DAG-folder modules are not reliably importable by every component. ## Lifecycle and cost, which the backend does not solve for you A custom backend creates an object per XCom, and Airflow's own cleanup — clearing a task instance, or `airflow db clean` — deletes only the database row. Nothing calls back into your backend to remove the object. You therefore need one of: bucket lifecycle rules that expire the prefix after N days, a scheduled cleanup DAG, or a key layout that makes reruns overwrite rather than accumulate. Otherwise the bucket grows monotonically and the storage bill becomes someone's surprise. There is also a latency cost: every push and every pull now performs a network round trip to object storage, on top of the database write. For thousands of small XComs — a dynamically mapped task with a large fan-out, say — that overhead is real. Some implementations offload only above a size threshold and let small values take the normal path. ## When it is actually worth it Use a custom backend when many teams legitimately pass medium-sized values, when you want the metadata database protected from payload growth without asking every DAG author to change code, or when you need XCom contents to sit inside a specific compliance boundary rather than in the Airflow database. Do not use it as a way to make gigabyte hops between tasks acceptable. The more common and often better answer is the convention version of the same idea: have tasks write to storage explicitly and push a URI, so the DAG shows a reader where the data went and the storage lifecycle is owned by the team that created it.