skip to content

A Django marketplace adds a router sending every read to a 'replica' alias for analytics; checkout now raises DoesNotExist reading back an order created inside transaction.atomic() - why, and how do you fix it?

level: seniorimportance: should knowfreq 40%

answer

  1. each query picks its own connection
  2. atomic() binds one alias
  3. uncommitted rows are invisible elsewhere
  4. scope the router, honour the instance
  5. in_atomic_block on the primary

basics

~20 s

The router sends the read to the replica connection, which is outside the transaction that atomic() opened on 'default', so the uncommitted order is invisible there. Scope the router to analytics models and keep reads on the primary inside transactions or with using('default').

solid answer

~30 s

Django picks a connection per query: the create goes through `db_for_write` to `'default'`, the `get()` through `db_for_read` to `'replica'`. `transaction.atomic()` without `using` opens a transaction on `'default'` only, so the read runs on a different connection that cannot see the uncommitted row; even after commit, replication lag can hide it. Fix it in Django's routing: have `db_for_read` return `'replica'` only for the analytics app's models and `None` otherwise, return `'default'` while `connections['default'].in_atomic_block` is true, and honour the `instance` hint. For a read that must be fresh outside a transaction, use `Order.objects.using('default')`.

code

python · 27 lines
python
from django.db import connections


class MarketplaceRouter:
    replica_apps = {'analytics'}

    def db_for_read(self, model, **hints):
        if connections['default'].in_atomic_block:
            return 'default'
        instance = hints.get('instance')
        if instance is not None and instance._state.db:
            return instance._state.db
        if model._meta.app_label in self.replica_apps:
            return 'replica'
        return None

    def db_for_write(self, model, **hints):
        return 'default'

    def allow_relation(self, obj1, obj2, **hints):
        pool = {'default', 'replica'}
        if obj1._state.db in pool and obj2._state.db in pool:
            return True
        return None

    def allow_migrate(self, db, app_label, model_name=None, **hints):
        return db != 'replica'

go deeper

for a junior

Recall that Django chooses a database for each query, and that a router can send reads and writes to different aliases.

for a middle

Explain why atomic() without using covers only 'default', and why a read on another connection cannot see rows that transaction has not committed.

for a senior

Diagnose the two failure modes separately, uncommitted visibility and post-commit lag, and fix the router's scope rather than sprinkling using() through views.

for a principal

Decide which models may ever be read stale, and whether the replica's saved load is worth routing rules every team must keep in their heads.

## What the router actually does A Django **database router** is consulted per query, not per request or per transaction. With this router installed: ```python class ReplicaRouter: def db_for_read(self, model, **hints): return 'replica' def db_for_write(self, model, **hints): return 'default' ``` checkout code such as this fails: ```python with transaction.atomic(): order = Order.objects.create(buyer=buyer, total=total) order = Order.objects.select_related('buyer').get(pk=order.pk) # DoesNotExist ``` 1. `Order.objects.create()` is a write, so `db_for_write` sends the `INSERT` to `'default'`. 2. `transaction.atomic()` with no `using` argument opened its transaction on the `'default'` connection only. Atomic blocks are per alias; nothing is opened on `'replica'`. 3. `Order.objects.get()` is a read, so `db_for_read` sends the `SELECT` to `'replica'`, a separate connection to a separate server. 4. The `INSERT` is not committed yet, so no other connection can see it, and the replica has not received it anyway. The `get()` raises `Order.DoesNotExist`. The router has no idea a transaction is open; Django's documentation says as much about the sample primary/replica router, which ignores both **replication lag** and transactions. After the block commits the same code can still fail for a moment, because the replica applies the change after the primary does. That second failure is lag; the first one would happen with zero lag. ## Confirming the diagnosis Before changing the router, prove where each statement went: 1. Turn on the `django.db.backends` logger at `DEBUG` (it logs only when `DEBUG` is true or a debug cursor is forced). Each SQL line ends with `alias=...`, so the `INSERT` shows `alias=default` and the failing `SELECT` shows `alias=replica`. 2. In a shell, check `Order.objects.all().db`: it returns the alias the router picks for a read without sending any SQL. 3. Repeat the failing code with `Order.objects.using('default').get(...)`. If it now succeeds inside the block, the cause is routing, not the write. ## Fix 1: route only what should go to the replica The real requirement was 'analytics reads on the replica', not 'every read on the replica'. Make the router answer only for the models that tolerate stale data and return `None` for everything else, so checkout, accounts and the admin keep reading from `'default'`. ## Fix 2: keep reads on the primary inside a transaction Each alias's connection wrapper exposes `in_atomic_block`, and `django.db.connections` gives each thread its own wrappers, so a router can check whether the current code is inside `atomic()` on the primary and send reads there. Be aware of the cost: - With `ATOMIC_REQUESTS` enabled on `'default'`, every view runs inside a transaction, so this rule sends every read in a view to the primary and the replica only serves management commands and background jobs. - `select_for_update()`, `get_or_create()` and `update_or_create()` already route through `db_for_write`, so they are not the problem. ## Fix 3: honour the instance hint and use() the primary on purpose - When code follows a relation from an object loaded on `'default'`, Django passes that object as `hints['instance']`. Returning `instance._state.db` keeps related reads on the same database, which is what Django does with no router at all. - For a single read that must be fresh, such as the order confirmation page right after checkout, bind it explicitly: `Order.objects.using('default').get(pk=pk)`. A manual alias always outranks the router. ## Comparing the fixes | Fix | Cures uncommitted read in atomic() | Cures lag after commit | Replica load kept | |---|---|---|---| | Scope `db_for_read` to analytics models | yes, for checkout models | yes, for checkout models | analytics only, as intended | | `in_atomic_block` check | yes | no | reads outside transactions | | `instance` hint | related reads only | related reads only | most reads | | `using('default')` at the call site | yes | yes | everything else | Keeping a user on the primary for a while after their own write, across requests, is a pinning policy; the routing layer is where a Django project would read a request-scoped flag for it, but the policy itself is a design choice beyond this router. ## Checklist for replica routers - Return an alias only for models you have decided may be stale. - Never route writes or locking reads to a replica. - Return `False` from `allow_migrate` for the replica alias. - Allow relations between objects from `'default'` and `'replica'` in `allow_relation`, since they hold the same data. - Test with a real replica or a delay, because a replica configured as a test mirror of `'default'` hides lag entirely.

  • Would transaction.atomic(using='replica') around the checkout read fix the DoesNotExist?
    No. It opens a second, separate transaction on the replica connection. The order's `INSERT` is still uncommitted on `'default'`, and the replica has not received it, so the read still misses. The read has to run on the connection that holds the open transaction, which means `'default'`.
  • Why do select_for_update() queries not suffer from this router, even though they are SELECTs?
    `select_for_update()` marks the QuerySet for writing, so Django asks `db_for_write`, which returns `'default'`. `get_or_create()` and `update_or_create()` are marked the same way. Only plain reads go through `db_for_read`, which is where the replica-everything rule does its damage.

saying these in an interview costs you the question

  • atomic() opens a transaction on every alias in DATABASES.
  • Django defers the INSERT until the atomic block commits, so the row does not exist yet.
  • The failure is only replication lag, so a faster replica fixes it.
  • Wrapping the read in atomic(using='replica') makes it see the new order.
  • A router is consulted once per request, so reads and writes share a connection.