In Celery, how do you send slow video-transcoding tasks to their own queue and pin a dedicated worker to consume only that queue?

middlemust knowfreq 50%

answer

separate lanes, not one line
a name-to-queue mapping in config
a per-call override also exists
the worker chooses its queues at start

basics

~20 s

Map the transcode task to a queue such as transcode in task_routes, or pass queue= to apply_async, then start a worker with -Q transcode. Unrouted tasks keep going to the default queue, named celery, which other workers drain.

solid answer

~40 s

By default every Celery task lands on one queue, celery (the task_default_queue setting), so a burst of long transcodes can occupy every worker process while password-reset emails wait behind them. I add task_routes = {'video.tasks.transcode': {'queue': 'transcode'}}; because task_create_missing_queues is on by default, Celery declares the transcode queue the first time it is used. For a one-off I can override per call with transcode.apply_async(args=(42,), queue='transcode'). Then I run two worker groups: celery -A video worker -Q transcode for the slow lane and celery -A video worker -Q celery for everything fast, and size each one separately. Routing only decides where a message waits; -Q decides who drains it, so a routed queue with no worker listening simply fills up.

By default every Celery task lands on one queue, `celery` (the `task_default_queue` setting), so a burst of long transcodes can occupy every worker process while password-reset emails wait behind them. I add `task_routes = {'video.tasks.transcode': {'queue': 'transcode'}}`; because `task_create_missing_queues` is on by default, Celery declares the `transcode` queue the first time it is used. For a one-off I can override per call with `transcode.apply_async(args=(42,), queue='transcode')`. Then I run two worker groups: `celery -A video worker -Q transcode` for the slow lane and `celery -A video worker -Q celery` for everything fast, and size each one separately. Routing only decides where a message waits; `-Q` decides who drains it, so a routed queue with no worker listening simply fills up.

code

python · 21 lines

python

from celery import Celery

app = Celery('video', broker='amqp://guest@localhost//')
app.conf.task_routes = {
    'video.tasks.transcode': {'queue': 'transcode'},
}


@app.task(name='video.tasks.transcode')
def transcode(video_id, profile):
    ...


@app.task(name='accounts.tasks.send_password_reset')
def send_password_reset(user_id):
    ...


transcode.delay(42, '1080p')        # task_routes -> queue 'transcode'
send_password_reset.delay(7)        # no route -> default queue 'celery'
transcode.apply_async(args=(43, '480p'), queue='transcode')  # explicit per call

go deeper

for a junior

Remember three names: task_routes (or queue=) sends a task to a queue, the default queue is called celery, and worker -Q chooses which queues a worker consumes.

for a middle

Explain that routing is decided at send time, that missing queues are declared automatically by default, and that a worker without -Q only drains task_queues or the default queue.

for a senior

Show the failure you are preventing: long transcodes holding every process while resets wait. Mention the silent pile-up when a routed queue has no consumer, and per-lane sizing and deploys.

for a principal

Frame queues as the unit of isolation and capacity: each lane gets its own worker fleet, scaling signal and alert, and routing belongs in config, not scattered through call sites.

Why one shared queue starves fast tasks

A queue in Celery is a named waiting line on the broker (RabbitMQ, Redis or Amazon SQS). A worker is a process started with celery -A <app> worker that takes messages off one or more queues and runs the tasks they describe. Out of the box there is exactly one queue, named celery (the value of task_default_queue), and every worker consumes it.

Picture a video platform: a transcode task that takes 20 minutes per upload, and a send_password_reset task that takes 200 ms. If 200 uploads arrive at once, every worker process is soon busy transcoding. A user who asks for a password reset now waits for a transcode to finish somewhere before their email is even picked up. The fast task is not slow; it is stuck behind slow work in the same line.

The fix is to give each class of work its own queue and its own workers.

Step 1 — route the task to a named queue

Celery decides a task's destination when it is sent, not when it runs. There are three ways to name the queue:

task_routes — a setting that maps task names (or glob patterns such as 'video.tasks.*') to a route, for example {'queue': 'transcode'}. This keeps routing in configuration, which is where the Celery docs recommend it.
queue= on the task decorator — @app.task(queue='transcode') sets a default on the task itself.
queue= on the call — transcode.apply_async(args=(42,), queue='transcode') overrides everything else for that one message.

A task with none of these goes to task_default_queue.

Keys in task_routes are registered task names, which default to the module path plus the function name (video.tasks.transcode), not the bare function name. A glob key such as 'video.tasks.*' routes a whole module at once, which is convenient for the transcode family, but it also catches any fast helper task someone later adds to the same module. For a lane that must stay slow-only, list the heavy tasks by exact name, or keep them in a module of their own. A route value can be a dict or simply the queue name as a string: {'video.tasks.transcode': 'transcode'} means the same thing.

Step 2 — let Celery declare the queue

task_create_missing_queues is enabled by default. When a route names a queue that is not listed in task_queues, Celery creates a declaration for it on the fly: a queue called transcode, bound to a direct exchange also called transcode, with routing key transcode. Transports without exchanges, such as Redis and SQS, still work because the exchange and queue share a name. Declaring queues explicitly with task_queues is an option for later, not a requirement.

Step 3 — pin workers with `-Q`

The worker's -Q (--queues) option lists the queues it consumes, comma-separated:

celery -A video worker -Q transcode -n transcode@%h — only transcodes.
celery -A video worker -Q celery -n fast@%h — only the default queue, where password resets and notifications land.
Optionally -Q transcode,celery on the slow machines, so they help with fast work when transcodes run dry.

A worker started without -Q consumes every queue in task_queues, which by default means only celery. The companion option -X (--exclude-queues) removes named queues from that set instead.

The resulting layout

queue	tasks routed there	consumed by
`transcode`	`video.tasks.transcode`	`worker -Q transcode` on the heavy machines
`celery` (default)	password resets, notifications, anything unrouted	`worker -Q celery` on small machines

Each group can now be sized, deployed and scaled on its own. How many processes each worker runs, and how many messages it prefetches, is a separate tuning question.

Pitfalls worth naming in an interview

A routed queue with no consumer fills up silently. Adding a route without starting a -Q transcode worker leaves messages waiting and their results PENDING.
Route names must match the registered task name exactly (video.tasks.transcode), not the Python function name.
The call-time argument wins. An apply_async(queue=...) in one code path can quietly bypass the configured route.
Priorities are not a substitute. Reordering within one queue still leaves fast tasks behind slow ones already running.
celery worker -A video is the removed 4.x order; in Celery 5 -A is a global option and must come before worker.

Separate queues are cheap isolation: one line of task_routes and one extra worker command.

## Why one shared queue starves fast tasks A **queue** in Celery is a named waiting line on the **broker** (RabbitMQ, Redis or Amazon SQS). A **worker** is a process started with `celery -A <app> worker` that takes messages off one or more queues and runs the tasks they describe. Out of the box there is exactly one queue, named `celery` (the value of `task_default_queue`), and every worker consumes it. Picture a video platform: a `transcode` task that takes 20 minutes per upload, and a `send_password_reset` task that takes 200 ms. If 200 uploads arrive at once, every worker process is soon busy transcoding. A user who asks for a password reset now waits for a transcode to finish somewhere before their email is even picked up. The fast task is not slow; it is **stuck behind** slow work in the same line. The fix is to give each class of work its own queue and its own workers. ## Step 1 — route the task to a named queue Celery decides a task's destination when it is **sent**, not when it runs. There are three ways to name the queue: - **`task_routes`** — a setting that maps task names (or glob patterns such as `'video.tasks.*'`) to a route, for example `{'queue': 'transcode'}`. This keeps routing in configuration, which is where the Celery docs recommend it. - **`queue=` on the task decorator** — `@app.task(queue='transcode')` sets a default on the task itself. - **`queue=` on the call** — `transcode.apply_async(args=(42,), queue='transcode')` overrides everything else for that one message. A task with none of these goes to `task_default_queue`. Keys in `task_routes` are **registered task names**, which default to the module path plus the function name (`video.tasks.transcode`), not the bare function name. A glob key such as `'video.tasks.*'` routes a whole module at once, which is convenient for the transcode family, but it also catches any fast helper task someone later adds to the same module. For a lane that must stay slow-only, list the heavy tasks by exact name, or keep them in a module of their own. A route value can be a dict or simply the queue name as a string: `{'video.tasks.transcode': 'transcode'}` means the same thing. ## Step 2 — let Celery declare the queue `task_create_missing_queues` is **enabled by default**. When a route names a queue that is not listed in `task_queues`, Celery creates a declaration for it on the fly: a queue called `transcode`, bound to a `direct` exchange also called `transcode`, with routing key `transcode`. Transports without exchanges, such as Redis and SQS, still work because the exchange and queue share a name. Declaring queues explicitly with `task_queues` is an option for later, not a requirement. ## Step 3 — pin workers with `-Q` The worker's `-Q` (`--queues`) option lists the queues it consumes, comma-separated: 1. `celery -A video worker -Q transcode -n transcode@%h` — only transcodes. 2. `celery -A video worker -Q celery -n fast@%h` — only the default queue, where password resets and notifications land. 3. Optionally `-Q transcode,celery` on the slow machines, so they help with fast work when transcodes run dry. A worker started **without** `-Q` consumes every queue in `task_queues`, which by default means only `celery`. The companion option `-X` (`--exclude-queues`) removes named queues from that set instead. ## The resulting layout | queue | tasks routed there | consumed by | |---|---|---| | `transcode` | `video.tasks.transcode` | `worker -Q transcode` on the heavy machines | | `celery` (default) | password resets, notifications, anything unrouted | `worker -Q celery` on small machines | Each group can now be sized, deployed and scaled on its own. How many processes each worker runs, and how many messages it prefetches, is a separate tuning question. ## Pitfalls worth naming in an interview - **A routed queue with no consumer fills up silently.** Adding a route without starting a `-Q transcode` worker leaves messages waiting and their results `PENDING`. - **Route names must match the registered task name** exactly (`video.tasks.transcode`), not the Python function name. - **The call-time argument wins.** An `apply_async(queue=...)` in one code path can quietly bypass the configured route. - **Priorities are not a substitute.** Reordering within one queue still leaves fast tasks behind slow ones already running. - **`celery worker -A video` is the removed 4.x order**; in Celery 5 `-A` is a global option and must come before `worker`. Separate queues are cheap isolation: one line of `task_routes` and one extra worker command.

What happens if you add the transcode route but forget to start a worker with -Q transcode?
The producer still declares the `transcode` queue and publishes into it, so messages accumulate on the broker with nobody consuming them. Their `AsyncResult` stays `PENDING` and nothing errors. A plain worker without `-Q` only drains the queues in `task_queues`, which by default is just `celery`.
Can a slow-lane worker also help with the default queue, and is the reverse a good idea?
Yes: `-Q transcode,celery` lets heavy machines pick up fast work when idle. The reverse, letting fast-lane workers take transcodes, reintroduces the starvation you routed away from, because a fast worker busy for 20 minutes is no longer fast.

A supermarket express lane: shoppers with one item are sent to their own till, and a cashier assigned to that till never serves full trolleys. Signposting the lane is routing; staffing it is -Q.

saying these in an interview costs you the question

Adding task_routes alone makes tasks run on a separate worker
The default Celery queue is called default
A worker without -Q consumes every queue that exists on the broker
You must declare every queue in task_queues before routing to it
celery worker -A proj -Q transcode is the Celery 5 command order

In Celery, how do you send slow video-transcoding tasks to their own queue and pin a dedicated worker to consume only that queue?

§ answer

basics

solid answer

code

§ go deeper

▾level notes4 levels

▾deep dive~4 min

Why one shared queue starves fast tasks

Step 1 — route the task to a named queue

Step 2 — let Celery declare the queue

Step 3 — pin workers with -Q

The resulting layout

Pitfalls worth naming in an interview

▾common follow-ups2 questions

▾analogy~10 s

▾red flags5 flags

Why one shared queue starves fast tasks

Step 1 — route the task to a named queue

Step 2 — let Celery declare the queue

Step 3 — pin workers with -Q

The resulting layout

Pitfalls worth naming in an interview

answer

go deeper

Step 3 — pin workers with `-Q`

Step 3 — pin workers with `-Q`