Celery Wiring

## What "wiring" means **Celery** is a separate task-queue library with its own worker processes. It needs no Django-specific package any more; you connect it to a Django project with a few lines of configuration so that: - the worker process can load Django (settings, models, the ORM), - Celery reads its options from Django's `settings.py`, - tasks defined in each Django app are found and registered, - code running inside Django (views, signals) can queue tasks. ## The celery.py module For a project laid out as `shop/manage.py` and `shop/shop/settings.py`, the recommended place is `shop/shop/celery.py`: ```python import os from celery import Celery os.environ.setdefault("DJANGO_SETTINGS_MODULE", "shop.settings") app = Celery("shop") app.config_from_object("django.conf:settings", namespace="CELERY") app.autodiscover_tasks() ``` Line by line: 1. **`DJANGO_SETTINGS_MODULE`** — the `celery` command is not `manage.py`, so nothing else tells it which settings to load. It must be set **before** `Celery(...)` is created: Celery installs its Django integration (the "fixup") at app creation only when this environment variable is present. `setdefault` keeps an explicit value from the environment. 2. **`Celery("shop")`** — the single Celery application for the project. Task names still default to the module path of each function, such as `orders.tasks.send_order_confirmation`. 3. **`config_from_object("django.conf:settings", namespace="CELERY")`** — use Django's settings as Celery's configuration. The string form means the worker does not need to serialise the settings object to child processes. The namespace means Celery reads **uppercase names prefixed with `CELERY_`**: `broker_url` becomes `CELERY_BROKER_URL`. 4. **`autodiscover_tasks()`** — with no arguments, Celery asks its Django integration for every installed app's config and imports `<app>.tasks` from each. Discovery is lazy by default and happens when the worker imports its modules. ## The project package's __init__.py ```python from .celery import app as celery_app __all__ = ("celery_app",) ``` This import makes the Celery app exist as soon as Django imports the project package. Without it, a web process might import a module using `@shared_task` before any app exists, and the task would not be bound to your configured app. ## Where tasks and settings go | Piece | Location | |---|---| | Celery app | `shop/celery.py` | | App import at startup | `shop/__init__.py` | | Celery options | `settings.py`, as `CELERY_BROKER_URL`, `CELERY_RESULT_BACKEND`, `CELERY_TASK_TIME_LIMIT`, ... | | Tasks | each app's `tasks.py`, decorated with `@shared_task` | | Worker | separate process: `celery -A shop worker -l INFO` | ## Order-confirmation example ```python # orders/tasks.py from celery import shared_task from django.core.mail import send_mail from .models import Order @shared_task def send_order_confirmation(order_id): order = Order.objects.select_related("customer").get(pk=order_id) send_mail( f"Order {order.pk} confirmed", "Thanks for your order.", None, [order.customer.email], ) ``` The view queues it with `send_order_confirmation.delay(order.pk)`. ## Common wiring mistakes | Mistake | Symptom | |---|---| | `DJANGO_SETTINGS_MODULE` set after `Celery(...)` | Worker cannot import models; no `delay_on_commit()` | | Celery options without the `CELERY_` prefix | Options silently ignored; defaults used | | Task defined outside the app's `tasks.py` | Worker reports an unregistered task | | No import in the project package's `__init__.py` | Web process queues through an unconfigured default app | | Worker started with other settings than the web process | Tasks read another database or broker | The last one is easy to miss in deployment: the web server and the worker are two processes with two environments, so the settings module and secrets must be set for both. ## What the Django integration does for you Once `DJANGO_SETTINGS_MODULE` is set, Celery's Django fixup calls `django.setup()` before task modules are imported in the worker, closes (or periodically recycles) database connections around each task so stale connections are not reused, and, unless you configured your own task base class, makes tasks use a Django-aware class that adds `delay_on_commit()`. These are the reasons the order of lines in `celery.py` matters.