A Django view creates an order in a transaction and queues a Celery email task that intermittently raises Order.DoesNotExist; what is wrong?

## Two processes, two connections In a Django project using Celery, the **web process** and the **worker** are separate processes with separate database connections. A database transaction's changes are only visible to other connections after it **commits**. Django runs in **autocommit** by default, but many projects wrap requests in transactions with `ATOMIC_REQUESTS = True`, or use `transaction.atomic()` around checkout logic. Inside such a block: 1. `Order.objects.create(...)` inserts the row — visible only to the view's connection. 2. `send_order_confirmation.delay(order.pk)` sends a message to the broker **right now**. 3. The view continues; the transaction commits when the block or request ends. A worker can pick the message up between steps 2 and 3. It runs `Order.objects.get(pk=order_id)` on its own connection, the row is not visible yet, and the task raises `Order.DoesNotExist`. Under light load the view usually wins and everything works, which is why the bug is **intermittent** and often only appears in production. ## The second failure: rollback If something later in the view raises, the transaction rolls back and the order never exists — but the message was already sent. At best the worker gets `DoesNotExist`; at worst, if the task only needed the id and an email address passed along, a customer receives a confirmation for an order that failed. | Queue call placement | Worker may miss the row | Email for rolled-back order | |---|---|---| | `delay()` inside the transaction | Yes | Yes | | `delay()` after the `atomic` block closes, no outer transaction | No | No | | `transaction.on_commit(...)` | No | No — callback discarded on rollback | | `delay_on_commit()` (Celery 5.4+) | No | No | ## The fix Django's `transaction.on_commit(func)` registers a callback that runs after the current transaction commits and is discarded if it rolls back. With no transaction active, it runs the callback immediately. Bind the arguments with `functools.partial` (or a lambda): ```python from functools import partial from django.db import transaction from .tasks import send_order_confirmation def place_order(cart): with transaction.atomic(): order = cart.to_order() transaction.on_commit(partial(send_order_confirmation.delay, order.pk)) return order ``` Celery 5.4 added a shortcut. When Celery's Django integration is active and no custom task base class is configured, tasks use Celery's Django task class, which provides: - `delay_on_commit(*args, **kwargs)` — `delay()` wrapped in `on_commit`; - `apply_async_on_commit(*args, **kwargs)` — the same for `apply_async()`. ```python send_order_confirmation.delay_on_commit(order.pk) ``` Two differences from `delay()` matter: - It returns **`None`**, not an `AsyncResult`: the message has not been sent, so there is no task id yet. If you must store the id, call `delay()` from your own `on_commit` callback. - If the project uses a custom task base class, it must inherit from the Django task class to get these methods. ## What on_commit does not solve - The message send can still fail after commit (broker down); the order exists but no email is queued. Guaranteeing both needs an outbox-style record written in the same transaction. - Tests using Django's `TestCase` wrap each test in a transaction that never commits, so on-commit callbacks do not run unless the test captures them explicitly. ## Diagnosis checklist 1. Is `ATOMIC_REQUESTS` on, or is the queue call inside `atomic()`? 2. Does the task fail with `DoesNotExist` only under load? 3. Replace `delay()` with `delay_on_commit()` or `on_commit(partial(...))`. 4. Keep passing the primary key, and let the task treat a missing row as a normal outcome.