In Celery 5.5 and later, what changes when a RabbitMQ payouts queue becomes a quorum queue, and how does native delayed delivery keep countdown tasks working?

## What changed in 5.5 Celery 5.5 added support for RabbitMQ **quorum queues**, RabbitMQ's replicated queue type, and alongside it **native delayed delivery** so that tasks with a `countdown` or `eta` keep working on them. Celery 5.6 keeps the same design. For a payouts service the attraction is plain: a payout message sitting on a replicated queue survives the loss of one RabbitMQ node. A queue becomes a quorum queue in one of two ways: - declare it with the queue argument `x-queue-type` set to `quorum` (a kombu `Queue` in `task_queues`); - or set `task_default_queue_type = "quorum"` so Celery's default queue is declared that way. The setting was added in 5.5 and defaults to `"classic"`. Celery's RabbitMQ guide pairs this with `broker_transport_options = {"confirm_publish": True}` so publishes wait for the broker's confirmation. ## Global QoS goes away Quorum queues do not support **global QoS**, the channel-wide prefetch limit a Celery worker normally adjusts at runtime. With `worker_detect_quorum_queues` at its default `True`, the worker checks its queues when it starts consuming over the `amqp` transport and, if any is a quorum queue, stops applying QoS globally. The prefetch count is then set once and stays put, which has knock-on effects: 1. `--autoscale` no longer works, because it grows and shrinks prefetch as it adds and removes processes. 2. `worker_enable_prefetch_count_reduction` becomes a no-op. 3. ETA and countdown tasks become a problem, described next. ## Why countdown tasks need help On a classic queue, a message with a countdown is fetched immediately and held in the worker until due; the worker bumps its prefetch count so the held message does not stop it fetching other work. With static prefetch that bump is impossible, so a held payout would occupy a slot until its countdown ends, and enough of them would stall the worker. Holding a message unacknowledged for a long time also collides with RabbitMQ's consumer acknowledgement timeout, which closes the channel with `PRECONDITION_FAILED` when a countdown outlives it. ## How native delayed delivery works When quorum queues are detected, Celery enables **native delayed delivery**, a chain of TTL queues that lets RabbitMQ itself do the waiting: - The worker declares 28 topic exchanges and queues, `celery_delayed_0` to `celery_delayed_27`. Level *n* has a message TTL of 2^n seconds and dead-letters into the level below; the last level dead-letters into the `celery_delayed_delivery` exchange. - At publish time, `apply_async(countdown=...)` encodes the delay as binary digits in the routing key and publishes to `celery_delayed_27`. A level whose bit is 1 holds the message in its queue for its TTL; a level whose bit is 0 passes it straight to the next exchange, so it emerges after roughly the requested delay. - The worker binds each of its queues' exchanges to `celery_delayed_delivery`, so the matured message is forwarded to the real queue and consumed normally. The message waits inside RabbitMQ, not in worker memory, so it holds no worker slot and no unacknowledged delivery. The queue type for these delay queues is `broker_native_delayed_delivery_queue_type`, default `"quorum"`. ## The trap: direct exchanges Native delayed delivery rewrites the routing key, which only a **topic** or **fanout** exchange can route. Celery's `task_default_exchange_type` defaults to `"direct"`. When a countdown task targets a direct exchange while quorum queues are in use, Celery logs "Direct exchanges are not supported with native delayed delivery" and publishes normally, and the task may block a worker until its ETA. That is why Celery's own example sets `task_default_exchange_type = "topic"`. ## Migrating an existing queue A queue's type is fixed when RabbitMQ first declares it, so an existing classic `payouts` queue cannot simply be redeclared as quorum under the same name; Celery's guide points to RabbitMQ's own migration documentation for moving from classic mirrored queues. In practice teams drain the old queue or route new work to a new quorum queue, and upgrade every worker to 5.5 or later first, since older workers know nothing about quorum-queue detection or native delayed delivery. ## Checklist for a payouts move to quorum queues | Item | Setting or effect | |---|---| | Make queues quorum | `x-queue-type: quorum` or `task_default_queue_type = "quorum"` | | Keep detection on | `worker_detect_quorum_queues = True` (default) | | Exchange for delayed tasks | `task_default_exchange_type = "topic"` | | Publisher confirms | `broker_transport_options = {"confirm_publish": True}` | | Lost features | `--autoscale`, prefetch-count reduction | The senior point is that quorum queues are not a drop-in switch: they change prefetch behaviour and move countdown waiting into the broker, and a default direct exchange quietly undoes the second half.