skip to content

Brokers and Result Backends

2 roadmaps6 questions

RabbitMQ, Redis or SQS as the broker, and Redis, a database or RPC as the result backend, each with its own durability. Asked because the choice decides whether a task can vanish.

on this pageshow

explore

questions

6

In Celery, what is the difference between the broker and the result backend, and does every app need a result backend?

level: juniorimportance: must knowfreq 62%

answer

  1. two stores, two different jobs
  2. messages in, state and return values out
  3. result_backend has no default
  4. get() needs somewhere to read from

basics

~20 s

Celery's broker carries task messages from the caller to the workers; the result backend stores each task's state and return value. The broker is required; the result backend is optional and unset by default, so fire-and-forget tasks need none.

solid answer

~40 s

The **broker** (`broker_url`, such as `amqp://`, `redis://` or `sqs://`) is the queue: `delay()` publishes a message there and a worker consumes it. The **result backend** (`result_backend`) is a separate store where the worker writes the task's state (`SUCCESS`, `FAILURE` and so on) and return value, so a caller can read them later through `AsyncResult`. `result_backend` has no default, so Celery falls back to its disabled backend: tasks still run, but `AsyncResult.get()` raises `NotImplementedError` ("No result backend is configured") and a chord cannot start. A payouts task that records its outcome in the application's own database needs no result backend; you add one when some caller or workflow actually reads results.

code

python · 15 lines
python
from celery import Celery

# broker only: no result_backend is set
app = Celery("payouts", broker="redis://localhost:6379/0")


@app.task
def send_payout(payout_id):
    # pays out and marks the Payout row as paid in the app's own database
    ...


result = send_payout.delay(42)  # publishes the message, returns an AsyncResult
print(result.id)                # the task id is still available
# result.get() would raise NotImplementedError: No result backend is configured.

go deeper

for a junior

Recall the two roles: the broker queues messages, the result backend stores states and return values. Know that only the broker is mandatory.

for a middle

Explain what the disabled backend does: tasks run, get() raises NotImplementedError, chords refuse to start. Name broker_url and result_backend and their defaults.

for a senior

Show you would only add a backend when something reads it, and that you weigh its write load, expiry and availability against using the app's own database as the record.

for a principal

Frame the result backend as an optional second system of record, and argue for keeping business outcomes in the domain database so Celery stays a transport.

## Two stores with two jobs A Celery deployment talks to up to two external systems, and they are easy to blur because the same Redis server often plays both roles. - The **broker** is the message transport. When code calls `send_payout.delay(42)`, Celery serialises the task name and arguments into a message and publishes it to a queue on the broker. Workers consume from that queue. Celery reads the broker's address from `broker_url`; its scheme picks the transport (`amqp://` for RabbitMQ, `redis://` or `rediss://` for Redis, `sqs://` for Amazon SQS). - The **result backend** is a store for outcomes. When a worker finishes a task, it can write the task's **state** (such as `STARTED`, `SUCCESS`, `FAILURE`, `RETRY` or `REVOKED`) and its **return value** or exception, keyed by the task id. Celery reads its address from `result_backend`. The broker answers "what work is waiting?"; the result backend answers "what happened to task X?". A message leaves the broker once a worker has acknowledged it, so the broker is not where results live. ## The broker is required Without a broker there is nothing to publish to, so a Celery app always has one. If `broker_url` is not set, Celery uses its documented default of `amqp://`, which means a RabbitMQ server on localhost with the default guest account: fine on a laptop, surprising in a container that has no RabbitMQ next to it. ## The result backend is optional `result_backend` has **no default**. When it is unset, Celery loads its `DisabledBackend` (it reports itself as `disabled://`). That backend silently drops every store call and raises on every read: 1. `task.delay()` and `apply_async()` still publish the message and still return an `AsyncResult` carrying the task id. 2. The worker runs the task normally; `store_result` is a no-op, so the outcome goes nowhere. 3. Calling `AsyncResult.get()` raises `NotImplementedError` with the message "No result backend is configured." 4. Starting a **chord** raises as well, because a chord's callback needs a backend to count finished header tasks. So "no result backend" is a legitimate, common configuration for fire-and-forget work, not a misconfiguration. ## Where each one is configured The two roles have separate setting families, which is a useful hint that they are separate systems: - `broker_url` chooses the transport, and `broker_transport_options` tunes it (for example the Redis `visibility_timeout`). - `result_backend` chooses the store, `result_backend_transport_options` tunes a Redis backend, and `result_expires` decides how long stored results live. - Per task, `ignore_result=True` skips storing the result even when a backend exists. A worker needs the broker settings to receive work and the backend settings to record outcomes; a web process that only sends tasks and never reads results needs only the broker settings. When both roles do point at one Redis server, keeping them on different database numbers makes it easy to tell queued messages from stored results while debugging. ## Deciding for a payouts service Consider a payouts service whose `send_payout` task calls a payment provider and then marks the `Payout` row as paid. The row in the application's own database is already the record of truth. A web page that shows payout status reads that row, not Celery's result. | Situation | Result backend needed? | |---|---| | Task writes its outcome to the app's own database | No | | A caller blocks on `AsyncResult.get()` for a return value | Yes | | A status endpoint reads `AsyncResult(task_id).state` | Yes | | The task is a chord header or callback | Yes | | Only failures matter, and they are logged or alerted | Usually no | Adding a backend has costs: every task writes a record, results must be expired (`result_expires`, one day by default), and the backend becomes one more system that can be down. Tasks that nobody reads can opt out individually with `ignore_result=True` even when a backend exists. ## Common mix-ups - Treating the backend as the queue: queued messages wait on the **broker**; the backend only ever sees outcomes. - Assuming results are kept in the broker when no backend is set: they are discarded. - Assuming both must be the same product: RabbitMQ as broker with Redis as result backend is a common pairing, and the same Redis server on two database numbers works too. - Assuming a missing backend stops tasks from running: the worker executes them exactly as before. The practical rule: configure a broker always, and a result backend only when some code will read what it stores.

  • Can one Redis server be both the Celery broker and the result backend?
    Yes, and it is common, often on different database numbers such as `redis://host:6379/0` for `broker_url` and `/1` for `result_backend`. The roles stay separate: the broker holds queued messages, the backend holds `celery-task-meta-<task_id>` keys that expire after `result_expires`. The cost is a shared failure domain: one Redis outage stops both publishing tasks and reading results.
  • What does ignore_result=True change once a result backend is configured?
    It stops that task writing its state and return value to the backend, which saves a write per call for tasks nobody reads. The `AsyncResult` returned by `delay()` is marked as ignored, so `get()` on it returns `None` immediately instead of waiting. It changes nothing when no backend is configured, because nothing would have been stored anyway.

A post office and a tracking database: the post office moves the parcel, the tracking database records whether it arrived. You can post a parcel without tracking; you just cannot ask where it is afterwards.

saying these in an interview costs you the question

  • The result backend is where queued task messages wait for a worker.
  • Without a result backend, delay() fails and the task never runs.
  • Celery keeps results in the broker when no result backend is set.
  • Every Celery task needs a result backend to report success.
  • Redis as the result backend forces Redis as the broker too.
open as a page

A payouts team on Celery's Redis broker raises visibility_timeout to 12 hours so delayed payouts stop running twice; what does that fix, and what does it cost?

level: seniorimportance: must knowfreq 40%

basics

~20 s

On Celery's Redis or SQS broker, a message left unacknowledged past visibility_timeout is redelivered, so long countdowns run twice. Twelve hours stops that, but a killed worker's tasks then wait up to 12 hours for redelivery.

open as a page

In Celery, what does AsyncResult.get() do to the caller waiting on it, and what do its timeout and propagate arguments and forget() control?

level: middleimportance: should knowfreq 45%

basics

~20 s

Celery's AsyncResult.get() blocks the caller until the task finishes, forever by default. timeout raises TimeoutError without stopping the task; propagate re-raises the task's exception in the caller; forget() deletes the stored result from the backend.

open as a page

For a Celery payouts service, how does broker_url choose between RabbitMQ, Redis and Amazon SQS, and what does each transport trade away?

level: middleimportance: should knowfreq 48%

basics

~20 s

The scheme of Celery's broker_url picks the kombu transport: amqp:// for RabbitMQ, redis:// for Redis, sqs:// for Amazon SQS. RabbitMQ is a native broker; Redis emulates acknowledgements with a visibility timeout; SQS is managed but has no events or remote control.

open as a page

In Celery, how do the Redis, database and rpc:// result backends differ in who can read a task's result, and for how long?

level: middleimportance: should knowfreq 38%

basics

~20 s

Celery's Redis and database backends store results any process can read by task id; Redis expires them after result_expires (one day), a database only when celery.backend_cleanup runs from beat. rpc:// sends results as messages, readable once and only by the sending client.

open as a page

In Celery 5.5 and later, what changes when a RabbitMQ payouts queue becomes a quorum queue, and how does native delayed delivery keep countdown tasks working?

level: seniorimportance: nice to knowfreq 20%

basics

~20 s

Celery 5.5+ detects RabbitMQ quorum queues and disables global QoS, so prefetch becomes static and --autoscale stops working. Countdown tasks would then block workers, so Celery enables native delayed delivery: RabbitMQ holds the message until due, which needs a non-direct exchange.

open as a page