A Celery app routes transcodes to a new 'transcode' queue, and they pile up unprocessed while other tasks run; why, and what does declaring task_queues change?

seniorshould knowfreq 30%

answer

declared is not consumed
which queues a bare worker drains
auto-creation hides typos
turn auto-creation off
queue type for auto-created queues

basics

~20 s

Routing creates the queue, but no worker consumes it: a worker without -Q drains only task_queues, by default just celery. Declaring task_queues and disabling task_create_missing_queues makes bare workers consume every declared queue and turns typos into errors.

solid answer

~40 s

With task_create_missing_queues on (the default), the producer declares transcode and publishes into it, so the messages sit on the broker. A worker started without -Q consumes the queues in task_queues, which by default is only celery, so nothing drains transcode and results stay PENDING. The immediate fix is a worker with -Q transcode, or add_consumer at runtime. The durable fix is to declare the topology with kombu Queue and Exchange objects in task_queues and set task_create_missing_queues = False: a bare worker then consumes every declared queue, a typo in a route raises QueueNotFound at send time, and -Q with an unknown name stops the worker at startup. In Celery 5.6, auto-created queues are classic unless task_create_missing_queue_type is 'quorum'.

With `task_create_missing_queues` on (the default), the producer declares `transcode` and publishes into it, so the messages sit on the broker. A worker started without `-Q` consumes the queues in `task_queues`, which by default is only `celery`, so nothing drains `transcode` and results stay `PENDING`. The immediate fix is a worker with `-Q transcode`, or `add_consumer` at runtime. The durable fix is to declare the topology with kombu `Queue` and `Exchange` objects in `task_queues` and set `task_create_missing_queues = False`: a bare worker then consumes every declared queue, a typo in a route raises `QueueNotFound` at send time, and `-Q` with an unknown name stops the worker at startup. In Celery 5.6, auto-created queues are classic unless `task_create_missing_queue_type` is `'quorum'`.

code

python · 13 lines

python

from kombu import Exchange, Queue

app.conf.task_default_queue = 'default'
app.conf.task_queues = (
    Queue('default', Exchange('default'), routing_key='default'),
    Queue('transcode', Exchange('transcode'), routing_key='transcode'),
    Queue('notifications', Exchange('notifications'), routing_key='notifications'),
)
app.conf.task_create_missing_queues = False
app.conf.task_routes = {'video.tasks.transcode': {'queue': 'transcode'}}

# A typo is now an error in the sender, not a stranded message:
# transcode.apply_async(args=(42,), queue='transcod')  -> raises QueueNotFound

go deeper

for a junior

Remember that a queue needs a worker listening to it; routing a task somewhere does not start anything that reads from there.

for a middle

Explain which queues a worker consumes without -Q, how auto-creation works, and what QueueNotFound and ImproperlyConfigured tell you after auto-creation is turned off.

for a senior

Diagnose from evidence: queue depth with zero consumers, inspect active_queues, worker start commands. Then harden with declared task_queues and explicit -Q on every worker.

for a principal

Treat queue topology as declared infrastructure: explicit queues, types and consumers reviewed together, so a new route cannot silently create an unowned queue.

What is actually happening

Two separate things must be true for a Celery task to run: its message must be published to a queue, and some worker must be consuming that queue. Routing only guarantees the first.

With the default task_create_missing_queues = True, a route to transcode that is not listed in task_queues makes Celery build a declaration on the fly: queue transcode, a direct exchange named transcode, routing key transcode. The producer declares it as it publishes, so the queue exists on the broker and the messages wait in it.

Now look at the workers. The Celery workers guide says a worker consumes all queues defined in task_queues, falling back to the default queue celery when that setting is unset. A fleet started as plain celery -A video worker therefore consumes only celery. The transcodes pile up, their AsyncResult.state stays PENDING, and nothing logs an error: from Celery's point of view everything is working.

Diagnosing it

Compare the queue's message count on the broker with its consumer count — the telltale sign is a growing count and zero consumers.
Ask the workers which queues they consume with celery -A video inspect active_queues.
Check the worker start commands for -Q, and the settings for task_queues.
Check the result side last: PENDING means no state has been stored for the task yet, which is exactly what an unconsumed message looks like, so it is a symptom rather than a separate fault.

The quick fix

Start a consumer for the queue: celery -A video worker -Q transcode, or tell running workers to add it with celery -A video control add_consumer transcode. Runtime changes are lost when the worker restarts, so the start command must change too.

The durable fix: declare the topology

Auto-creation is convenient, but it turns every spelling mistake into a new, unconsumed queue: queue='transcod' succeeds and the message is stranded. Declaring queues explicitly closes that gap:

List every queue in task_queues as kombu Queue objects, each with its Exchange and routing key.
Include the default queue there too — the docs require task_default_queue to be listed once task_queues is set.
Set task_create_missing_queues = False.

What changes afterwards:

situation	auto-creation on (default)	`task_queues` declared, auto-creation off
route or `queue=` names an unknown queue	new queue created, message stranded	`QueueNotFound` raised at send time
worker started without `-Q`	consumes `task_queues`, by default only `celery`	consumes every declared queue
`worker -Q` names an unknown queue	queue created for that worker	worker refuses to start (`ImproperlyConfigured`)
queue arguments such as priority	Celery defaults	set per `Queue`

QueueNotFound subclasses KeyError, so a typo fails loudly in the web request that sent it instead of disappearing. Because a bare worker now drains everything, the fast-lane workers need -Q celery or -X transcode (--exclude-queues) to stay fast.

Queue types in 5.5 and 5.6

RabbitMQ offers classic and quorum queues (a replicated type). Celery now lets you choose which one it declares:

task_default_queue_type (added in 5.5, default 'classic') sets the type of the default queue.
task_create_missing_queue_type (added in 5.6, default 'classic') sets the type of queues Celery auto-creates; 'quorum' adds x-queue-type: quorum. Only 'classic' and 'quorum' are accepted.
task_create_missing_queue_exchange_type (5.6, default None) sets the exchange type for those auto-created queues.
Queues you declare yourself in task_queues ignore the two 5.6 settings; set queue_arguments={'x-queue-type': 'quorum'} on the Queue instead.

Before 5.6, a team that had standardised on quorum queues could still end up with a classic transcode queue simply because it was auto-created. What quorum queues cost and require on the broker side is a broker-configuration question rather than a routing one.

Takeaways

The general lesson is that Celery separates two decisions that feel like one: producers choose where a message waits, and each worker's start command chooses what it drains. Nothing checks that the two agree, so the deployment has to.

A queue with messages and no consumer is a common routing outage, and it is silent.
Declared topology plus task_create_missing_queues = False converts silent strandings into errors.
Always start workers with an explicit -Q so the deployment, not a default, states who drains what.

## What is actually happening Two separate things must be true for a Celery task to run: its message must be **published to a queue**, and some worker must be **consuming that queue**. Routing only guarantees the first. With the default `task_create_missing_queues = True`, a route to `transcode` that is not listed in `task_queues` makes Celery build a declaration on the fly: queue `transcode`, a `direct` exchange named `transcode`, routing key `transcode`. The producer declares it as it publishes, so the queue exists on the broker and the messages wait in it. Now look at the workers. The Celery workers guide says a worker consumes **all queues defined in `task_queues`**, falling back to the default queue `celery` when that setting is unset. A fleet started as plain `celery -A video worker` therefore consumes only `celery`. The transcodes pile up, their `AsyncResult.state` stays `PENDING`, and nothing logs an error: from Celery's point of view everything is working. ## Diagnosing it - Compare the queue's message count on the broker with its consumer count — the telltale sign is a growing count and zero consumers. - Ask the workers which queues they consume with `celery -A video inspect active_queues`. - Check the worker start commands for `-Q`, and the settings for `task_queues`. - Check the result side last: `PENDING` means no state has been stored for the task yet, which is exactly what an unconsumed message looks like, so it is a symptom rather than a separate fault. ## The quick fix Start a consumer for the queue: `celery -A video worker -Q transcode`, or tell running workers to add it with `celery -A video control add_consumer transcode`. Runtime changes are lost when the worker restarts, so the start command must change too. ## The durable fix: declare the topology Auto-creation is convenient, but it turns every spelling mistake into a new, unconsumed queue: `queue='transcod'` succeeds and the message is stranded. Declaring queues explicitly closes that gap: 1. List every queue in `task_queues` as kombu `Queue` objects, each with its `Exchange` and routing key. 2. Include the default queue there too — the docs require `task_default_queue` to be listed once `task_queues` is set. 3. Set `task_create_missing_queues = False`. What changes afterwards: | situation | auto-creation on (default) | `task_queues` declared, auto-creation off | |---|---|---| | route or `queue=` names an unknown queue | new queue created, message stranded | `QueueNotFound` raised at send time | | worker started without `-Q` | consumes `task_queues`, by default only `celery` | consumes every declared queue | | `worker -Q` names an unknown queue | queue created for that worker | worker refuses to start (`ImproperlyConfigured`) | | queue arguments such as priority | Celery defaults | set per `Queue` | `QueueNotFound` subclasses `KeyError`, so a typo fails loudly in the web request that sent it instead of disappearing. Because a bare worker now drains everything, the fast-lane workers need `-Q celery` or `-X transcode` (`--exclude-queues`) to stay fast. ## Queue types in 5.5 and 5.6 RabbitMQ offers **classic** and **quorum** queues (a replicated type). Celery now lets you choose which one it declares: - `task_default_queue_type` (added in 5.5, default `'classic'`) sets the type of the default queue. - `task_create_missing_queue_type` (added in 5.6, default `'classic'`) sets the type of queues Celery **auto-creates**; `'quorum'` adds `x-queue-type: quorum`. Only `'classic'` and `'quorum'` are accepted. - `task_create_missing_queue_exchange_type` (5.6, default `None`) sets the exchange type for those auto-created queues. - Queues you declare yourself in `task_queues` ignore the two 5.6 settings; set `queue_arguments={'x-queue-type': 'quorum'}` on the `Queue` instead. Before 5.6, a team that had standardised on quorum queues could still end up with a classic `transcode` queue simply because it was auto-created. What quorum queues cost and require on the broker side is a broker-configuration question rather than a routing one. ## Takeaways The general lesson is that Celery separates two decisions that feel like one: producers choose where a message **waits**, and each worker's start command chooses what it **drains**. Nothing checks that the two agree, so the deployment has to. - A queue with messages and no consumer is a common routing outage, and it is silent. - Declared topology plus `task_create_missing_queues = False` converts silent strandings into errors. - Always start workers with an explicit `-Q` so the deployment, not a default, states who drains what.

After you declare task_queues, why might your fast-lane workers suddenly start taking transcodes?
A worker without `-Q` consumes every queue in `task_queues`. Once `transcode` is declared there, a bare `celery -A video worker` drains it too. Give fast workers `-Q default,notifications` or `-X transcode` so the lanes stay separate.
What does task_create_missing_queue_type change, and what does it leave alone?
Added in 5.6, it sets the RabbitMQ queue type Celery uses when it auto-creates a missing queue: `'classic'` by default, or `'quorum'`. It does not affect queues you declare yourself in `task_queues`, nor the default queue, whose type comes from `task_default_queue_type` (5.5).

saying these in an interview costs you the question

If a routed queue has no consumer, Celery raises an error when sending
A plain celery worker drains every queue that exists on the broker
Setting task_create_missing_queues to False deletes auto-created queues from the broker
task_create_missing_queue_type also changes queues declared in task_queues
Tasks stuck in PENDING always mean the result backend is broken

A Celery app routes transcodes to a new 'transcode' queue, and they pile up unprocessed while other tasks run; why, and what does declaring task_queues change?

§ answer

basics

solid answer

code

§ go deeper

▾level notes4 levels

▾deep dive~4 min

What is actually happening

Diagnosing it

The quick fix

The durable fix: declare the topology

Queue types in 5.5 and 5.6

Takeaways

▾common follow-ups2 questions

▾red flags5 flags

What is actually happening

Diagnosing it

The quick fix

The durable fix: declare the topology

Queue types in 5.5 and 5.6

Takeaways

answer

go deeper