Airflow task instances sit in the queued state for hours and never start — how do you diagnose it?

## What queued means An Airflow task instance moves `scheduled` → `queued` → `running`. `queued` is the boundary between the scheduler's world and the execution world: the scheduler has verified upstream dependencies, the trigger rule, pool availability and concurrency limits, and has handed the task to the executor. A task stuck in `queued` therefore is *never* a dependency problem — it is a capacity, routing or infrastructure problem. ## Step 1: are Airflow's own limits binding? The most common and most benign cause is that the task is legitimately waiting for a slot. - **Pool exhausted.** Open the Pools page: if used slots equal total slots, everything else assigned to that pool queues. A task with `pool_slots=4` in a 4-slot pool waits for the pool to be completely empty. - **DAG limits.** `max_active_tasks` and `max_active_runs` on the DAG cap it independently of cluster capacity. - **`[core] parallelism`.** The scheduler-wide ceiling. The tell for all of these is queued tasks *while workers are idle*. ## Step 2: is the hand-off actually reaching a worker? If limits are not binding, the message went somewhere nobody is listening. With **`CeleryExecutor`**, check the task instance's `queue` value in the UI, then check what your workers subscribe to. `airflow celery worker --queues gpu` is a *subscription*, not a validated assignment: a task routed to a queue that no running worker consumes waits forever with no error anywhere. This happens after renaming a queue, decommissioning a worker pool, or a typo in `queue=`. Next, confirm workers are alive and connected — Flower, `celery inspect active`, or simply the worker logs — and that the broker is reachable and not out of memory. A Redis broker that has evicted keys, or a RabbitMQ node in an alarm state, will silently drop or stall messages. With **`KubernetesExecutor`**, the diagnosis moves to the cluster: `kubectl get pods -n <airflow-ns>` and `kubectl describe` the worker pod. `Pending` with `FailedScheduling` means insufficient CPU/memory, an unschedulable node selector or an untolerated taint, or a namespace ResourceQuota that is full. `ErrImagePull`/`ImagePullBackOff` means the worker image tag or registry credentials are wrong. If no pod exists at all, the scheduler could not create it — look for Kubernetes API errors or RBAC denials in the scheduler log. ## Step 3: is the scheduler healthy? If the scheduler died after queueing tasks, nothing will progress them. On restart, Airflow tries to adopt or re-queue orphaned task instances; if adoption fails you can see tasks queued against an executor that no longer knows about them. Check the scheduler's heartbeat, its logs for exceptions in the loop, and — in Celery deployments — whether the executor believes it has more running slots than it does. Clearing the affected task instances forces a clean re-dispatch and is the standard remedy once the root cause is fixed. ## Step 4: the metadata database A saturated or locked metadata database slows the scheduling loop enough that tasks appear stuck. Long-running queries, missing autovacuum on PostgreSQL, or an undersized instance all show up as an Airflow that "queues but does not run". ## Guardrails so it does not happen silently - Alert on queued-state duration, not just on failure. A task queued for an hour is an incident even though nothing has failed. - Recent Airflow 2 releases will fail or retry a task that has been queued past a configured timeout, which converts a silent hang into a visible failure — useful, but make sure you still capture the reason. - Validate `queue` names in CI against the queues your workers actually subscribe to; this single check eliminates the most common Celery instance of the problem. - Monitor broker depth and worker count together: rising depth with flat worker count is the signature of an under-scaled or dead fleet.