Background Jobs & Scheduling

If beat was down, each overdue entry is sent once on restart, not once per missed slot. If workers were down, beat kept sending and messages queued. An expires option drops stale copies; beat_cron_starting_deadline skips crontab runs that are too late.

A Celery chord needs a result backend because something must record each header task's completion and result before the body can start. If one header task fails, the others still run, but the body never runs and is marked FAILURE with a ChordError.

revoke only broadcasts the id to workers, which skip that task when they reach it; a running task continues unless terminate=True signals its pool process. Revoked ids live in worker memory, so a full restart forgets them unless workers use --statedb.

SIGTERM starts a warm shutdown that waits, unbounded, for running tasks, so the later SIGKILL loses them. Cold shutdown (SIGQUIT) cancels them; soft shutdown (5.5+) first waits worker_soft_shutdown_timeout. Use REMAP_SIGTERM=SIGQUIT, a timeout under the grace period, and acks_late.

Use self.retry() for transient failures: a new message runs the task again. Raise Reject for a message that should leave the queue, dead-lettered or requeued, which needs acks_late. Raise Ignore when the work is moot: it acks and records nothing.

Routing creates the queue, but no worker consumes it: a worker without -Q drains only task_queues, by default just celery. Declaring task_queues and disabling task_create_missing_queues makes bare workers consume every declared queue and turns typos into errors.

On RabbitMQ, priority is ignored unless the queue was declared with x-max-priority, and higher numbers go first. On Redis, kombu emulates it with separate lists where 0 is served first, so 9 is the lowest priority.

apply_async turns countdown into an eta and publishes at once. On Redis or a classic RabbitMQ queue a worker takes the message immediately and holds it unacknowledged in memory until due, so hours-long waits pile up and can run twice.

Celery's rate_limit is a token bucket kept separately by each worker instance for each task type, not a cluster-wide limit. Four workers at '10/m' allow about 40 starts a minute; tasks over the limit wait inside the worker rather than failing.

Split queues by latency and resource class, not per task: a fast lane for resets and notifications, a heavy lane for transcodes, maybe a bulk lane. Each extra queue buys isolation but costs a worker fleet, scaling rules and alerts.

Split Celery into two worker deployments on separate queues: prefork reports with concurrency bounded by cores and RAM, prefetch 1 and child recycling; gevent webhooks with hundreds of greenlets and a larger prefetch. Scale mostly by adding workers.

In Celery, a group sends one message per call, so 10,000 photos means 10,000 parallel tasks. starmap() sends one message and runs every call in sequence inside one task. chunks(it, n) sends a group of starmap tasks, each running n calls in sequence.

Celery 5.5+ detects RabbitMQ quorum queues and disables global QoS, so prefetch becomes static and --autoscale stops working. Countdown tasks would then block workers, so Celery enables native delayed delivery: RabbitMQ holds the message until due, which needs a non-direct exchange.