Background Jobs & Scheduling

Celery's broker carries task messages from the caller to the workers; the result backend stores each task's state and return value. The broker is required; the result backend is optional and unset by default, so fire-and-forget tasks need none.

A bound Celery task calls raise self.retry(exc=exc, countdown=N) in its except block: Celery publishes a new message for the same task id, delayed N seconds (180 by default), up to max_retries (3), then re-raises exc as the failure.

delay(*args, **kwargs) is a shortcut for apply_async(args, kwargs) with no execution options. Both serialize the arguments, publish one message to the broker and return an AsyncResult with the task id at once; neither waits for a worker.

In Celery, a chain runs tasks in sequence and passes each result on; a group runs tasks in parallel; a chord is a group plus one body task that receives the list of all header results. Parallel thumbnails, then publish, is a chord.

Celery's AsyncResult.get() defaults to disable_sync_subtasks=True, so inside a worker task it raises RuntimeError: a task blocking on another can exhaust the pool and deadlock. Express the dependency with a chain, chord or link callback instead.

Celery sets no time limit by default, so a hung task holds its worker slot forever. soft_time_limit raises SoftTimeLimitExceeded inside the task so it can clean up; time_limit kills and replaces the pool process and fails the task with TimeLimitExceeded.

autoretry_for makes Celery call retry() when a listed exception escapes the task. retry_backoff sets the delay to factor x 2^retries seconds, capped by retry_backoff_max (600), and retry_jitter, on by default, draws a random delay between zero and the computed value.

Map the transcode task to a queue such as transcode in task_routes, or pass queue= to apply_async, then start a worker with -Q transcode. Unrouted tasks keep going to the default queue, named celery, which other workers drain.

Run CPU-heavy rendering on Celery's default prefork pool: separate processes each run their own interpreter and can keep a core busy. Run I/O-bound webhook calls on gevent or eventlet, where hundreds of greenlets wait on sockets cheaply inside one process.

The -B flag embeds a beat scheduler in every worker, so three containers run three independent schedulers and each sends the billing task at 02:00. Celery has no lock between beat processes: run exactly one standalone celery beat.

On Celery's Redis or SQS broker, a message left unacknowledged past visibility_timeout is redelivered, so long countdowns run twice. Twelve hours stops that, but a killed worker's tasks then wait up to 12 hours for redelivery.

Celery acknowledges a message just before running it by default; acks_late=True moves the ack to after the task finishes, so a worker crash mid-charge redelivers the message and the charge runs again. A killed pool child is still acked unless reject_on_worker_lost=True.

Each Celery worker reserves concurrency × worker_prefetch_multiplier messages (4 per slot by default), so webhooks wait in a busy worker's buffer. -O fair, the default since 4.0, only avoids busy children inside one worker; a lower prefetch fixes it.

A Celery signature is one task call packed as data: task name, args, kwargs and execution options. In a chain, a .s() signature gets the previous step's result prepended to its arguments; an immutable .si() signature ignores it.

celery -A proj inspect active asks every worker, over the broker, which tasks it is executing. inspect reserved lists tasks it prefetched but has not started, inspect scheduled the ETA or countdown tasks it holds; messages still queued appear in none.

Celery's --concurrency (-c, setting worker_concurrency) sets how many tasks one worker runs at once: child processes under prefork, threads or greenlets under other pools. Unset, it defaults to the number of CPUs the worker detects.

DatabaseScheduler builds beat's schedule from PeriodicTask rows and checks a one-row PeriodicTasks change marker, bumped by save and delete signals, about every 5 seconds. Bulk update() or raw SQL skips those signals, so beat notices only at its next full reload.

crontab fields are matched against the wall clock of the app's timezone; with timezone unset and enable_utc True, that is UTC, so hour=2 means 02:00 UTC. PersistentScheduler resets itself when a stored zone changes; DatabaseScheduler needs a manual reset.

Celery's AsyncResult.get() blocks the caller until the task finishes, forever by default. timeout raises TimeoutError without stopping the task; propagate re-raises the task's exception in the caller; forget() deletes the stored result from the backend.

The scheme of Celery's broker_url picks the kombu transport: amqp:// for RabbitMQ, redis:// for Redis, sqs:// for Amazon SQS. RabbitMQ is a native broker; Redis emulates acknowledgements with a visibility timeout; SQS is managed but has no events or remote control.

Celery's Redis and database backends store results any process can read by task id; Redis expires them after result_expires (one day), a database only when celery.backend_cleanup runs from beat. rpc:// sends results as messages, readable once and only by the sending client.

In Celery, a link callback is sent only after the task succeeds, with its return value prepended to the callback's arguments. A link_error errback runs when the task fails; a new-style errback receives the request, the exception and the traceback.

Task events are messages a Celery worker publishes as each task is received, started, succeeds or fails. They are off by default; celery worker -E turns them on. Flower builds its task list from that stream, so without events it shows no task history.

Celery writes STARTED only when task_track_started, or the task's track_started, is enabled, and it is off by default; PENDING just means the backend has no record of the id. Built-in states: PENDING, STARTED, RETRY, SUCCESS, FAILURE, REVOKED.

The apply_async(queue=) argument wins, then a queue option set on the task itself, then the first router in task_routes that returns a route. Within a task_routes dict an exact task name beats a glob pattern.

With bind=True, Celery passes the task object itself as the first argument, self. Through self.request a running task reads its execution context: task id, retry count, args and kwargs, worker hostname, delivery info, eta and expires.

Celery's default JSON carries strings, numbers, booleans, None, lists and dicts, plus datetime, Decimal, UUID and bytes, which kombu tags and restores; anything else raises EncodeError at the call. Pickle carries more but lets broker writers run code on workers.

A Celery message carries only the task's name, looked up in the worker's own registry. Names default to module path plus function name, so a worker that never imported the module, or imported it under another path, rejects it as unregistered.

Both recycle Celery prefork child processes: worker_max_tasks_per_child replaces a child after N tasks, worker_max_memory_per_child (kilobytes) replaces it after the task that crossed the limit finishes. Each restart costs a fork and warm-up.