Apache Airflow

Point the core xcom_backend setting at your class, which subclasses BaseXCom and overrides serialize_value and deserialize_value. Serialize writes the payload to external storage and returns a small pointer; deserialize resolves the pointer back into the value.

Split on schedule and ownership, not on size. Work that shares one data interval and one failure story belongs in one DAG, grouped with TaskGroups for readability; work on a different cadence or owned by another team belongs in its own DAG, linked explicitly.

Characterise the workload first. Many short tasks and stable dependencies favour CeleryExecutor's warm workers; heterogeneous resource needs, per-team images and strong isolation favour KubernetesExecutor's pod-per-task, which costs seconds of startup on every task.

Treat XCom as control-plane metadata only: small JSON facts and object URIs, never payloads and never secrets, since values are unencrypted and visible in the UI. Add scheduled retention, and enforce the rules with cluster policies rather than documentation alone.

Airflow 2 supports multiple active schedulers with no leader election. They coordinate through the metadata database, taking row-level locks with SELECT ... FOR UPDATE so only one scheduler can claim and queue a given task instance.

When the same integration plus its policy — auth, tagging, error mapping, a quality gate — is being retyped across many DAGs by teams who should not have to know it. Otherwise prefer provider operators and thin Python tasks over hooks: a custom operator library is a versioned product you then own.

Match the mechanism to who owns the dependency. Datasets suit a declared data contract between separately owned DAGs; TriggerDagRunOperator suits a producer that deliberately fans out to a downstream it owns; ExternalTaskSensor suits waiting on a DAG you cannot modify, and is the most fragile.