What policy would you set for XCom use across a shared multi-team Airflow platform?

## Frame the decision On a single-team Airflow, XCom hygiene is a code-review habit. On a shared platform with dozens of teams, it is a capacity and security question, because every XCom lands in one metadata database that every team's scheduling depends on. The policy question is therefore: what may pass through this shared channel, who enforces it, and what happens when someone ignores it. ## Rule one: size and shape Allowed payloads are facts about a run, not the run's output: a row count, a partition date, an external system's job id, an object URI, a short status string. Small, JSON-serializable, and — the property I would push hardest — *reconstructible*. If a downstream task can derive the S3 path from `data_interval_start` instead of reading it from an XCom, the pipeline survives clearing, retries and trimmed history. Forbidden are datasets. The reason is not aesthetic. XCom values are rows in the metadata database, so payload growth becomes scheduler-loop latency, slower UI, larger backups and longer restores. Those costs land on every team, not the one that caused them, which is the textbook definition of something a platform must govern rather than suggest. I would state an explicit ceiling — say, single-digit kilobytes as the norm and anything approaching a megabyte requiring a conversation — because a number is enforceable and "keep it small" is not. ## Rule two: no secrets, ever This one is non-negotiable and is the security answer interviewers listen for. Airflow encrypts Connections and Variables with its Fernet key. XCom values are **not** encrypted: they sit in the metadata database in the clear, appear in database backups, and are rendered in the web UI to anyone who can view that task instance. A task that fetches a short-lived token and pushes it for the next task to use has effectively published it. The pattern instead is that each task retrieves the credential from a Connection or the configured secrets backend at the moment it needs it. ## Rule three: retention is a platform job XCom rows do not expire. `airflow db clean --clean-before-timestamp` must run on a schedule with a retention window the platform chooses, and the `xcom` table's size should be a monitored metric with an alert, because table growth is the leading indicator that someone is passing payloads. Retention also interacts with correctness: once an old run's XComs are trimmed, clearing a single downstream task in that run leaves it with nothing to pull. That is another reason the policy prefers values that can be recomputed from the logical date. ## The genuine tradeoff: transparent backend versus explicit convention Here is where reasonable platform teams differ, and a principal-level answer should present both sides rather than declare one. **A custom XCom backend** that offloads payloads to object storage keeps `return df` working, protects the metadata database without asking anyone to rewrite DAGs, and is a single central change. Against it: it applies to every XCom in the deployment, so all teams inherit the round-trip latency; the DAG source no longer tells a reader where data went; one shared bucket blurs tenant isolation and creates a retention bill nobody owns, since Airflow's cleanup deletes rows and never the objects. **Explicit convention** — tasks write to their own storage and push a URI — keeps ownership, cost and lifecycle with the team that produced the data, and keeps the DAG honest about what it is doing. Against it: it is a change every team must make, and it is only as strong as your enforcement. My default is the convention, with the backend held in reserve for a specific, measured problem — for example, a compliance requirement that pipeline values not reside in the Airflow database. ## Enforcement beats documentation A wiki page does not stop a deadline. Airflow's cluster policies — `dag_policy` and `task_policy` defined in `airflow_local_settings.py` — run at parse time against every DAG and task in the deployment, and can reject or mutate what they see. That is where a platform expresses rules mechanically: reject tasks that opt into pushing when the team is on a deny list, force `do_xcom_push=False` on operator types known to emit noise, require a tag or an owner. Pair it with a CI lint on DAG repositories and a dashboard of the `xcom` table, and you have prevention, detection and a conversation trigger rather than a rule nobody reads. ## What I would actually write down One page: allowed content with examples; a size ceiling with a number; a hard ban on secrets with the reason; the retention window and who runs the clean; the convention for object paths derived from the data interval; and the escalation path for a team that thinks they are the exception. Then the cluster policy that enforces as much of it as parse time can see, and a metric that catches the rest.