skip to content

For a Celery payouts service, how does broker_url choose between RabbitMQ, Redis and Amazon SQS, and what does each transport trade away?

level: middleimportance: should knowfreq 48%

answer

  1. the URL scheme picks the transport
  2. a real broker versus an emulated one
  3. who holds unacknowledged messages
  4. SQS: polling, no events, no remote control

basics

~20 s

The scheme of Celery's broker_url picks the kombu transport: amqp:// for RabbitMQ, redis:// for Redis, sqs:// for Amazon SQS. RabbitMQ is a native broker; Redis emulates acknowledgements with a visibility timeout; SQS is managed but has no events or remote control.

solid answer

~40 s

Celery hands `broker_url` to kombu, and the scheme picks the transport: `amqp://` (the default) for RabbitMQ, `redis://` or `rediss://` for Redis, `sqs://` for Amazon SQS. **RabbitMQ** is a native AMQP broker: Celery declares durable queues, publishes persistent messages, and the broker itself tracks acknowledgements, so a dead worker's unacked messages return to the queue. **Redis** is a data store that kombu turns into a queue: unacked messages sit in a side structure and are restored only after `visibility_timeout` (3600 s by default), and durability is whatever the Redis server's persistence gives. **SQS** is fully managed, but the worker polls it, redelivery also runs on a visibility timeout (1800 s in kombu), and it has no events or remote control, so Flower and `celery inspect` do not work. For payouts, RabbitMQ is the conservative choice.

code

python · 12 lines
python
from celery import Celery

# RabbitMQ
app = Celery("payouts", broker="amqp://payouts:secret@rabbit:5672/payouts")

# Redis over TLS, with a longer visibility timeout for the Redis transport
# app.conf.broker_url = "rediss://:secret@redis:6379/0"
# app.conf.broker_transport_options = {"visibility_timeout": 7200}

# Amazon SQS with credentials from the host's IAM role
# app.conf.broker_url = "sqs://"
# app.conf.broker_transport_options = {"region": "eu-west-1"}

go deeper

for a junior

Recall the three URL schemes, amqp://, redis:// and sqs://, and that amqp:// is the default when nothing is set.

for a middle

Explain how each transport handles an unacknowledged message: protocol-level on RabbitMQ, visibility-timeout restore on Redis and SQS, with their default windows.

for a senior

Tie the choice to failure modes: lost Redis data, long visibility windows after a crash, and losing Flower and remote control on SQS.

for a principal

Weigh operating a broker cluster against the weaker guarantees and tooling gaps of Redis or SQS, given what a lost or delayed payout costs.

## The URL picks the transport Celery does not talk to brokers directly; it uses **kombu**, its messaging library, and `broker_url` tells kombu which **transport** to load. Only the scheme is required; the rest defaults per transport. - `amqp://user:pass@host:5672/vhost` selects RabbitMQ (through the `pyamqp` client). It is also Celery's documented default when `broker_url` is unset. - `redis://:password@host:6379/0` selects Redis; `rediss://` is the same over TLS. - `sqs://` selects Amazon SQS. With an IAM role on the host, the URL can be just `sqs://`; the default region is `us-east-1` unless `broker_transport_options` sets `region`. - Several URLs of the same transport, as a list or semicolon-separated, give failover under `broker_failover_strategy`. Per-transport knobs go in `broker_transport_options`, a plain dict whose keys belong to the chosen kombu transport. ## RabbitMQ: a native broker RabbitMQ implements queues, exchanges and acknowledgements itself. By default Celery declares its queues **durable** and publishes messages as **persistent** (`task_default_delivery_mode` is `persistent`), so queued payouts survive a broker restart. Acknowledgement is part of the protocol: if a worker's connection drops, the broker hands its unacknowledged messages to another consumer without waiting for a timer. Monitoring events and remote control are supported, and since Celery 5.5 so are RabbitMQ quorum queues. The cost is operating a separate broker cluster. ## Redis: a data store emulating a broker Redis has no acknowledgements, so kombu emulates them: 1. A queue is a Redis list; a worker pops a message off it. 2. The popped message is copied into an **unacked** hash plus a sorted index keyed by delivery time. 3. Acknowledging deletes that entry; a periodic scan restores any entry older than `visibility_timeout`, which defaults to **3600 seconds** in kombu's Redis transport. Consequences for a payouts service: a message whose worker was killed waits up to the visibility timeout before anyone runs it; a countdown or ETA longer than the timeout is redelivered while still waiting; and durability of queued messages is whatever persistence the Redis server is configured with. An eviction policy that removes keys can also delete kombu's queue-binding keys. Celery's docs add that Redis handles small messages well and that large ones can congest it. In exchange Redis is simple to run and is often already present as a cache or result backend. ## Amazon SQS: managed, with fewer features SQS removes broker operations entirely, but the transport is thinner: - Workers **poll**. Long polling is on by default (`wait_time_seconds`, 10 s), and an empty poll sleeps for `polling_interval` (1 s). - Redelivery is again timer-based: kombu creates queues with a `visibility_timeout` of **1800 seconds** unless you set one, and AWS caps it at 12 hours. With `predefined_queues`, you set it on the queue in AWS instead. - The transport supports only direct exchanges, so broadcast-style routing is out. - It emits no **events** and answers no **remote control**, so Flower, `celery events` and `celery inspect` / `celery control` do not work. - SQS is only a broker; results need a separate result backend such as Redis or a database. ## Side by side | Concern | RabbitMQ (`amqp://`) | Redis (`redis://`) | Amazon SQS (`sqs://`) | |---|---|---|---| | Unacked message after worker death | returned when the connection drops | restored after `visibility_timeout` (3600 s) | visible again after the queue's visibility timeout (1800 s from kombu) | | Durability of queued messages | durable queues, persistent messages | depends on Redis persistence | managed by AWS | | Events / Flower | yes | yes | no | | Remote control (`inspect`, `control`) | yes | yes | no | | Operations burden | run a broker cluster | often already running | none | ## Choosing for payouts A payouts task moves money, so the question is which transport makes a queued payout hardest to lose and quickest to recover: 1. RabbitMQ gives protocol-level acknowledgements and durable, persistent messages, and keeps Flower and remote control. 2. Redis is acceptable if its persistence is configured for it and every countdown stays well under `visibility_timeout`. 3. SQS suits a team already on AWS that accepts polling and losing Flower and `celery inspect`. Whichever is chosen, a payout task must still tolerate running twice, because every transport can redeliver.

  • Why does a team on SQS lose Flower and celery inspect?
    Flower and `celery events` consume the event messages workers broadcast, and `celery inspect` / `celery control` send remote-control messages that every worker must receive. Celery's docs list SQS with no monitoring and no remote control support; the SQS transport routes only through direct exchanges, while events use a topic exchange and broadcast control uses fanout. Teams on SQS fall back to queue metrics from AWS and to logs.
  • Where do per-transport settings such as visibility_timeout go?
    In `broker_transport_options`, a dict passed straight to the kombu transport. Keys are transport-specific: `visibility_timeout` means the unacked restore window on Redis and the queue attribute on SQS, while `region`, `wait_time_seconds` and `predefined_queues` exist only for SQS. An unknown key is not a Celery setting, so a typo is not caught at startup.

saying these in an interview costs you the question

  • Redis tracks acknowledgements natively, just like RabbitMQ does.
  • A Redis broker returns a dead worker's message the instant its connection drops.
  • Flower works the same whichever transport broker_url selects.
  • Celery's default broker, with nothing configured, is Redis on localhost.
  • With SQS as the broker, AsyncResult.get() works without any result backend.