In Django, why can a background task enqueued inside transaction.atomic() fail to find the order it was given, and how does on_commit() fix it?

## The race A **background task** is work handed to a queue and executed later by a **worker** — usually a separate process with its own database connection. Django 6.0 added a Tasks framework (`django.tasks`) with `Task.enqueue()`; many projects use Celery. Either way the sequence inside a request looks like: 1. `Order.objects.create(...)` inserts the row inside your open transaction. 2. `send_receipt_task.enqueue(order_id=order.pk)` puts a message on the queue. 3. The worker picks it up — possibly within milliseconds. 4. Your view finishes other work and the outermost `atomic` block commits. If step 3 happens before step 4, the worker's `Order.objects.get(pk=...)` runs on a connection that cannot see your uncommitted insert and raises `Order.DoesNotExist`. If step 4 is a rollback instead, the worker either fails or — worse — acts on data that was never saved. ## The Django fix Register the enqueue as an after-commit callback: ```python from functools import partial from django.db import transaction with transaction.atomic(): order = Order.objects.create(email=email, total=total) transaction.on_commit(partial(send_receipt_task.enqueue, order_id=order.pk)) ``` - **`on_commit`** calls the callable only after the outermost block commits and discards it on rollback. - **`functools.partial`** binds the arguments, because callbacks are called with none; Django's Tasks documentation uses exactly this form. - **The primary key**, not the instance, is passed: the worker re-reads committed data, and task arguments generally must be serialisable anyway. ## A development-time blind spot The default `TASKS` setting in Django 6.x uses `ImmediateBackend`, which runs the task synchronously inside `enqueue()` — on the same connection, inside your transaction, so it *can* see the uncommitted row. Code that enqueues inside the block therefore works in development and breaks only when a real worker-based backend is configured. Django ships no worker; production backends come from third-party packages. Reviewing for "enqueue inside atomic" catches the bug before production does. ## What moves, and what it costs | Approach | Worker sees the row? | Rolled-back order triggers task? | Task lost if the process dies after commit? | |---|---|---|---| | `enqueue()` inside the block | Not guaranteed | Yes | No | | `on_commit(partial(task.enqueue, ...))` | Yes | No | Yes | | Outbox row in the same transaction, relayed later | Yes | No | No | The `on_commit` version is the right default for receipts, notifications and cache refreshes. For work that must never be lost — a payment capture, a ledger export — a row written in the same transaction and relayed by a separate process gives an at-least-once guarantee; that pattern is its own subject. ## Related pitfalls - **ATOMIC_REQUESTS views**: the view's transaction commits only when the view returns, so enqueueing anywhere in the view has the same race; `on_commit` still fixes it. - **Signals**: a `post_save` receiver that enqueues runs inside the caller's transaction too; wrap its enqueue in `on_commit` as well. - **Failure handling**: an exception raised by `enqueue()` inside a callback happens after the commit; decide whether to log it (`robust=True`) or let it surface. - **Retries in the worker** are not a fix: retrying until the row appears hides the race and still runs tasks for rolled-back orders. ## A review checklist 1. Search for `.enqueue(`, `.delay(` and similar calls, and check whether each runs inside `atomic` or an `ATOMIC_REQUESTS` view. 2. Wrap each such call in `transaction.on_commit(partial(...))`, binding primary keys. 3. Configure a worker-based backend in at least one pre-production environment so the race can actually surface. 4. For each task, write down whether losing it after a commit is acceptable; if not, it needs a persisted outbox.