How does Django's test --parallel option split the suite and the database, and what makes a suite unsafe to run with it?
answer
- one database per process
- numbered suffixes on the test database
- split by test case class
- shared resources outside the database
basics
~20 s--parallel runs test case classes in several processes, one per CPU core by default, and gives each process its own clone of the test database. Tests are unsafe when they share anything else: files, an external cache, ports or order.
solid answer
~40 s`manage.py test --parallel` (or `--parallel auto`) starts one worker per core from `multiprocessing.cpu_count()`, or as many as `DJANGO_TEST_PROCESSES` says; `--parallel 4` fixes the number. The runner creates the test database once, then clones it per worker as `test_shop_1`, `test_shop_2` and so on, so database state is isolated automatically. Work is split by **test case class**, and if there are fewer classes than workers Django starts fewer. What is not isolated is everything outside the database: shared `MEDIA_ROOT` files, a shared cache server, fixed ports, environment variables, and tests that quietly depend on another class running first. `SerializeMixin` with a `lockfile` forces classes that share a resource to run one at a time. Debugging needs `--parallel 1`, since `--pdb` is refused with more workers, and tracebacks need the `tblib` package to display properly.
go deeper
Know that --parallel runs tests in several processes and that each process gets its own copy of the test database.
Explain how the worker count is chosen, the _1, _2 clone naming, and why work is split by test case class.
Diagnose parallel-only flakiness from shared files, external caches, ports and order dependence, and fix it with isolation or SerializeMixin.
Plan suite structure and CI capacity for parallelism: class sizes, database connection limits and which resources need per-worker isolation.
## What `--parallel` does By default `manage.py test` runs every test in one process. `--parallel` spreads the suite across several processes so a multi-core machine is actually used. | Invocation | Number of workers | |---|---| | no option | 1 (serial) | | `--parallel` or `--parallel auto` | `DJANGO_TEST_PROCESSES` if set, else `multiprocessing.cpu_count()` | | `--parallel 4` | 4 | Two details reduce the count further: the runner never starts more workers than there are **test case classes**, and it needs a supported `multiprocessing` start method (fork, spawn, or since Django 6.0 forkserver). ## How work is divided The runner **partitions the suite by test case class**: all methods of `OrderTests` go to the same worker, in order, and different classes are distributed across workers. This keeps `setUpClass`, `setUpTestData` and class-level fixtures valid, because a class never straddles two processes. It also means one huge class limits the speed-up: a class with 400 slow tests runs on one core no matter how many you have. ## How the database is isolated 1. The runner creates the main test database as usual, for example `test_shop`, and migrates it. 2. It then **clones** it once per worker, adding a numeric suffix: `test_shop_1`, `test_shop_2`, and so on. On PostgreSQL a clone is created from the test database used as a template; with an in-memory SQLite database, forked workers inherit a copy, while spawn and forkserver workers get a copy written to disk. 3. Each worker's connection settings are switched to its own clone before it runs tests. Because every worker owns a database, `TestCase` rollbacks and `TransactionTestCase` flushes in one worker cannot disturb another. With `--keepdb`, the clones are kept too. ## What still collides The database is the only resource Django clones. Anything else shared between workers turns into flaky failures: - **Files.** Tests writing to the same `MEDIA_ROOT` or a fixed temporary path overwrite each other. Give each class its own temporary directory, or use `InMemoryStorage` for media. - **Caches and queues outside the process.** A Redis or Memcached server is shared by every worker; `LocMemCache` is per process and safe. - **Ports and external services.** Two `LiveServerTestCase` classes are fine because each binds a free port, but a test that starts a helper on a hard-coded port is not. - **Hidden order dependence.** A test that only passes after another class created some state breaks when the classes land in different workers. Running `--shuffle` in serial mode is a cheap way to smoke these out. When a group of classes genuinely must share a resource, mix in **`SerializeMixin`** and give them the same `lockfile` attribute; Django holds a file lock so those classes run one at a time even under `--parallel`. ```python import os from django.test import TestCase from django.test.testcases import SerializeMixin class ExportDirMixin(SerializeMixin): lockfile = os.path.abspath(__file__) class InvoiceExportTests(ExportDirMixin, TestCase): ... class ReportExportTests(ExportDirMixin, TestCase): ... ``` ## Operational notes - **Debugging:** `--pdb` raises an error with more than one worker; rerun the failing class with `--parallel 1`. - **Tracebacks:** failures travel between processes by pickling; install `tblib` so they display properly, and rerun serially when a traceback still cannot be shown. - **Database load:** N workers mean N open connections and N clones on the database server; check connection limits on shared CI databases. ## A worked sizing example A suite of 1,200 tests in 90 classes on an 8-core CI runner: 1. `--parallel` starts 8 workers and creates `test_shop` plus eight clones; `--timing` shows the cloning cost separately. 2. Classes are spread across the workers, so wall time approaches the slowest worker's share rather than the sum. 3. If one worker finishes much later than the rest, `--durations` usually points at one oversized class; splitting it rebalances the run. The speed-up is rarely a clean factor of eight, because of setup and cloning, uneven class sizes, and the database server itself becoming the bottleneck, but on suites dominated by database-backed tests it is usually the single largest win.
- A suite has 16 test case classes and one of them holds 70% of the tests; what does --parallel 8 achieve?Django partitions by test case class, so the giant class runs in a single worker and caps the speed-up; the other 15 classes finish early on the remaining workers. Splitting that class into several classes by feature is what lets `--parallel` use the cores.
- Why can tests that pass serially fail under --parallel even though each worker has its own database?Only the database is cloned. Tests that share files under one directory, a cache server, a hard-coded port or an ordering assumption across classes collide once classes run in different processes at the same time. Isolate the resource per class, or group the classes with `SerializeMixin` and a shared `lockfile`.
It is like giving each exam hall its own photocopy of the answer sheet: nobody can scribble on anyone else's copy, but if two halls share one whiteboard (a cache server or a folder), they still overwrite each other.
saying these in an interview costs you the question
- Believing all parallel workers share one test database with row locks
- Thinking Django distributes individual test methods, not test case classes, across workers
- Saying --parallel without a number means two processes
- Assuming a shared Redis cache is isolated per worker like the database
- Expecting --pdb to work normally with several workers