In Airflow, how do LocalExecutor, CeleryExecutor and KubernetesExecutor differ in where tasks run?

## What an executor actually is A common misconception is that the executor is a service you deploy. It is not. The executor is a Python class instantiated **inside the scheduler process**, chosen with `[core] executor` in `airflow.cfg` (or `AIRFLOW__CORE__EXECUTOR`). When the scheduler decides a task instance may run, it puts it in the `queued` state and hands it to the executor, whose only job is: get this task instance executed somewhere and report back what happened. Everything else — dependency logic, retries, state — belongs to the scheduler and the metadata database and is identical whichever executor you pick. That is why changing executors does not change your DAG code. ## SequentialExecutor One task at a time, in the scheduler process, and the only executor that works with a SQLite metadata database. It is the default in a fresh Airflow 2 install and exists so `airflow standalone` works on a laptop. It is not a production option, and Airflow 3 removed it entirely. ## LocalExecutor `LocalExecutor` forks a subprocess per task instance on the machine running the scheduler. It needs a real database (PostgreSQL or MySQL) but no broker, no cluster, no extra moving parts. Its ceiling is that machine's CPU and memory, bounded by `[core] parallelism`. For a single team with tens of DAGs and modest tasks — especially tasks that mostly submit work elsewhere, like a `BigQueryInsertJobOperator` or a `dbt` invocation — it is a perfectly serious production choice and the one that fails in the fewest ways. Its weaknesses are real: the scheduler host is also the execution host, so a memory-hungry task can starve scheduling; you cannot give one task a different Python environment; and scaling means a bigger machine. ## CeleryExecutor `CeleryExecutor` turns execution into a distributed queue. The executor publishes a message to a **broker** (`[celery] broker_url` — Redis or RabbitMQ), and `airflow celery worker` processes running elsewhere consume it. Each worker runs several tasks concurrently (`[celery] worker_concurrency`), and total capacity is roughly workers × concurrency, still bounded by `[core] parallelism` per scheduler. Two properties matter operationally. First, **queues**: a task can set `queue="gpu"` and you start workers with `airflow celery worker --queues gpu`, which routes heavy or specialised work to the right machines. Second, **code distribution**: every worker must have the same DAG files and the same Python dependencies as the scheduler, usually via a baked image or git-sync. Version skew between scheduler and worker is the classic Celery deployment bug. The costs are a broker to operate and monitor, and workers that are running (and billed) whether or not there is work — though autoscaling on queue depth, e.g. with KEDA, softens that. ## KubernetesExecutor `KubernetesExecutor` creates **one pod per task instance** through the Kubernetes API. The pod runs `airflow tasks run` for that task, writes its state, and terminates. There are no idle workers: capacity is whatever the cluster can schedule. The big win is isolation and heterogeneity. Each task can have its own image, resource requests, node selector or tolerations, either from a pod template file or per task with `executor_config={"pod_override": k8s.V1Pod(...)}`. One task can ask for 16 GiB and a GPU node while its neighbour asks for 200 MiB. A task that leaks memory takes down only its own pod. The big cost is per-task overhead: scheduling a pod, pulling an image and booting a Python process takes seconds, which is fine for a ten-minute Spark submission and terrible for four hundred two-second tasks. You also need remote logging, because pod-local logs disappear with the pod. ## Hybrids `CeleryKubernetesExecutor` and `LocalKubernetesExecutor` let short tasks take the cheap path and heavy ones get their own pod. From Airflow 2.10, `[core] executor` accepts a comma-separated list of executors and a task can name which one it wants, which is the direction the project has taken. ## Choosing Start with `LocalExecutor` and move only when a real constraint appears: horizontal scale or queue routing pushes you to Celery; per-task isolation, wildly heterogeneous resources or a strong existing Kubernetes platform push you to Kubernetes.