How would you choose between CeleryExecutor and KubernetesExecutor for a team's Airflow platform?

## Frame it as a workload question, not a taste question The interviewer is testing whether you characterise before you choose. Four inputs decide it: task shape, isolation requirements, cost model, and the platform your organisation already operates. ## Task shape Count tasks and measure their durations. A DAG of 400 tasks that each take three seconds is the pathological case for `KubernetesExecutor`: pod scheduling plus image pull plus Python start can exceed the task's own runtime, so wall-clock time is dominated by overhead and the Kubernetes API takes a burst of create/delete calls. The same 400 tasks under `CeleryExecutor` land on warm workers with negligible overhead. Invert it for long tasks. A DAG of twelve tasks that each run for forty minutes barely notices ten seconds of pod startup, and gains isolation and exact resource sizing. ## Isolation and heterogeneity Ask what tasks need from their environment. If every task is Python against the same pinned dependency set, a shared worker image is simpler and cheaper. If one team needs a CUDA image, another needs an old pandas, and a third wants 32 GiB for one aggregation, `KubernetesExecutor` (or `KubernetesPodOperator` under any executor) is the mechanism that keeps them from fighting: per-task images, resource requests, node selectors and tolerations, and a memory leak that kills only its own pod. Under Celery, one task that exhausts a worker's memory takes down its co-tenants on that worker. Security boundaries follow the same logic: a pod per task can carry its own service account and network policy, which is far harder with long-lived shared workers. ## Cost model Celery workers cost money while idle. If your load is a nightly spike and eight quiet hours, that idle capacity is real waste — though it is fixable: run Celery workers on Kubernetes and autoscale replicas on broker queue depth (KEDA is the common tool), which recovers much of the elasticity while keeping warm-worker latency for the bulk of the day. Kubernetes workers cost only while running, but the per-task overhead is a cost too: at very high task counts you are paying for image pulls and pod churn, and you may need image pre-pulling or a node pool that stays warm anyway. ## Operational surface Celery adds a broker (Redis or RabbitMQ) that you must monitor, size and recover — a broker outage stops execution, and a Redis with the wrong eviction policy loses messages. It also demands strict artefact discipline: scheduler and every worker must carry identical DAG code and dependencies, or you get version-skew bugs that appear only on some tasks. Kubernetes adds cluster dependencies: RBAC for pod creation, quotas, pod templates, remote logging (mandatory, since pods vanish), and engineers who can read `kubectl describe pod` at 3 a.m. If that platform already exists and is well run, this is close to free; if Airflow would be the first serious Kubernetes workload, you have doubled the scope of the project. ## Do not choose only once Since Airflow 2.10, `[core] executor` accepts multiple executors and a task can name which one it uses; the older `CeleryKubernetesExecutor` and `LocalKubernetesExecutor` gave a coarser version of the same idea. The mature answer for a platform serving several teams is usually hybrid: short, homogeneous tasks on warm Celery workers, heavy or dependency-conflicting tasks on pods, and `KubernetesPodOperator` available to any team that would rather ship an image than a Python package. ## Also consider not choosing either For a single team with a few dozen DAGs whose tasks mostly submit work elsewhere — a warehouse query, a dbt run, a Spark job — `LocalExecutor` on one well-sized machine is a legitimate production answer with the fewest failure modes. Adopting a distributed executor because it sounds more serious is a real and common mistake. ## How to defend the decision State the measurements you would take before committing: task-count and duration distribution, peak concurrent tasks, memory profile of the worst tasks, current queued-time percentiles, and the cost of idle worker capacity. Then commit, name the migration path (the executor is a config change; the DAGs are not affected), and name the escape hatch — a hybrid configuration — if the workload turns out mixed.