skip to content

In a Rails 8.1 suite using parallelize(workers: :number_of_processors), how does each forked worker get its own database, and what shared state can still make tests flaky?

level: seniorimportance: should knowfreq 38%

answer

  1. forked processes, DRb test queue
  2. database name suffixed _0, _1, ...
  3. threshold 50, PARALLEL_WORKERS overrides
  4. parallelize_setup for files and caches
  5. threads: no per-worker databases

basics

~20 s

Rails forks one process per worker and, after each fork, appends the worker number to every test database name and loads the schema there; files, external caches and services stay shared, so isolate them in parallelize_setup.

solid answer

~40 s

`parallelize(workers: :number_of_processors)` (the default `with: :processes`) forks that many workers and feeds them tests over a DRb queue. Before forking Rails clears the connections; after forking, each worker appends `_<n>` to every test database name (`parking_test_0`, `parking_test_1`) and rebuilds it from the schema. Together with the per-test transaction rollback, that isolates database rows. It only kicks in above 50 tests (`threshold:` or `config.active_support.test_parallelization_threshold`), and `PARALLEL_WORKERS` overrides the count. What stays shared is everything outside those databases: files under `tmp/storage`, an external cache or Redis, ports, and third-party sandboxes. I give those per-worker names in `parallelize_setup { |worker| ... }` and clean up in `parallelize_teardown`. `with: :threads` creates no extra databases and skips those hooks.

code

bash · 5 lines
bash
# Force 8 workers on a CI box, whatever its core count
PARALLEL_WORKERS=8 bin/rails test

# Rule shared state in or out: one process, one database
PARALLEL_WORKERS=1 bin/rails test test/integration

go deeper

for a junior

Know that test_helper.rb runs tests in parallel processes by default and that each worker has its own copy of the test database.

for a middle

Explain the fork, rename to _n and schema-load sequence, the 50-test threshold and the PARALLEL_WORKERS override.

for a senior

Diagnose flaky parallel runs by ruling workers in or out, and isolate files, external caches and ports with parallelize_setup and parallel_worker_id.

for a principal

Weigh CI worker counts against database provisioning cost, and decide when a suite should own its database setup with parallelize_databases: false.

## What parallelize turns on The generated `test/test_helper.rb` calls **`parallelize(workers: :number_of_processors)`** inside `ActiveSupport::TestCase` (or adds `with: :threads` on platforms without `fork`). The method's defaults are `with: :processes`, a **threshold of 50 tests**, and one database per process. - `:number_of_processors` becomes the machine's available processor count. - `PARALLEL_WORKERS=4` in the environment overrides the `workers:` argument, and also forces parallel mode even below the threshold. - A count of 1 or fewer means no parallelism. - With 50 or fewer tests selected, the run prints `Running N tests in a single process (parallelization threshold is 50)`. Raise the threshold with `parallelize threshold: 100` or `config.active_support.test_parallelization_threshold`. ## How a worker gets its own database With `with: :processes`, Rails starts a DRb server holding the test queue and forks the workers: 1. **Before fork** - Active Record clears all connections, so no socket is shared across processes. 2. **Fork** - each worker is a separate Ruby process with its own memory, class-level state and `ENV`. 3. **After fork** - for every test database config, Rails renames the database to `"#{database}_#{n}"`, for example `parking_test_2` on PostgreSQL or `storage/test.sqlite3_2` on SQLite, and rebuilds it from the app's schema file, replicas included. 4. **Run** - each worker pulls tests from the queue. Inside a test, transactional tests still roll back that test's writes. 5. **Shutdown** - each worker runs the `parallelize_teardown` hooks with its number; since 8.1.4 the parent process also runs them once, with no number, after the workers exit. Rails 8.1 adds `ActiveSupport::TestCase.parallel_worker_id` (and an instance method) to read the worker number. It also adds `parallelize(..., parallelize_databases: false)`, also settable as `config.active_support.parallelize_test_databases = false`, for apps that manage test databases themselves. Rails' own comment warns that several processes on one database can deadlock. ## What is still shared The database is per worker, but much else is not: | Shared resource | Typical failure | Fix | |---|---|---| | Disk paths such as `tmp/storage` (the generated Active Storage test service root) | one worker's cleanup deletes files another is reading | per-worker directory chosen in `parallelize_setup` | | An external cache or Redis server | keys written by one worker are read or cleared by another | namespace by worker number | | Fixed ports, temp files with fixed names | address-in-use errors, overwritten files | derive names from the worker number | | Third-party sandboxes | unique-email or rate-limit collisions | unique data per worker, or stub the call | The generated test environment uses `:null_store` for `Rails.cache`, so the built-in cache is not a source of leaks until someone changes it. ```ruby class ActiveSupport::TestCase parallelize(workers: :number_of_processors) parallelize_setup do |worker| ActiveStorage::Blob.service.root = Rails.root.join("tmp/storage-#{worker}").to_s end parallelize_teardown do |worker| FileUtils.rm_rf(Rails.root.join("tmp/storage-#{worker}")) if worker end end ``` ## Threads instead of processes `parallelize(workers: :number_of_processors, with: :threads)` uses Minitest's thread executor in one process: - **no per-worker databases** are created, and `parallelize_setup` / `parallelize_teardown` do not run; - class-level state, globals and anything memoised at class level are shared by every thread; - Rails' documentation states it will not work with system tests. It suits JRuby and TruffleRuby, or suites without database-heavy tests. On CRuby, processes are the default. ## Diagnosing a flaky parallel suite - Re-run with `PARALLEL_WORKERS=1`. If the failure disappears, the cause is shared state across workers, not the code under test. - Look for anything the table above lists, then give it a worker-specific name in `parallelize_setup`. - If a test truly needs serial execution, for example one that spawns threads which open their own transactions, isolate it in a class with `self.use_transactional_tests = false` and clean up explicitly. Keep that class small. - Print `parallel_worker_id` in a failing test's message or log line so the CI output says which worker hit the problem. - Check whether the failing tests share a resource name rather than a database row. Two workers writing `tmp/export.csv` is the classic case, and it never reproduces on a single process. - Remember that `PARALLEL_WORKERS` on a CI box with many cores can create far more databases than the database server allows connections for. Size it against the server's connection limit, not just the core count.

  • Why does a Rails run of 30 selected tests not use the parallel workers?
    Rails only parallelises when more tests are selected than the threshold, 50 by default. A smaller run prints `Running 30 tests in a single process (parallelization threshold is 50)`. Setting `PARALLEL_WORKERS` forces parallel mode, and `parallelize threshold:` or `config.active_support.test_parallelization_threshold` changes the cut-off.
  • When would you set parallelize_databases: false in a Rails 8.1 test helper?
    When the app provisions its own test databases, for example a pre-built database per CI container, and Rails must not rename and rebuild them after each fork. You then take over isolation yourself, usually with a `parallelize_setup` hook that connects each worker to its own database. Several processes sharing one database can deadlock.

  • Rails parallel workers share one test database and rely only on transactions.
  • parallelize(workers: 8) always parallelises, even a run of five tests.
  • with: :threads gives each thread its own test database.
  • Forked workers share class-level variables, so a memoised class cache leaks between workers.
  • Per-worker databases also isolate files written under tmp/storage.

Assumes Rails 8.1: parallel_worker_id and parallelize(parallelize_databases: false) are new in 8.1; the threshold of 50 and per-worker databases predate it.