skip to content

With Minitest, why do tests run in random order, and how do you replay a failing order with --seed and shrink it with --bisect?

level: middleimportance: must knowfreq 50%

answer

  1. Run options: --seed N
  2. run_order defaults to :random
  3. --seed / SEED= replays the shuffle
  4. minitest --bisect finds the minimal set
  5. i_suck_and_my_tests_are_order_dependent!

basics

~20 s

Minitest shuffles classes and methods with a printed seed to expose tests that depend on each other. Rerun with --seed N to replay the order, then minitest --bisect with that seed to find the smallest failing set.

solid answer

~50 s

Each run picks a seed (or takes `--seed N` / `SEED=N`), prints it as `Run options: --seed N`, and calls `srand` with it. Every class's `run_order` defaults to `:random`, so Minitest shuffles the classes and the `test_` methods inside each class; a test that only passes after another has left state behind fails on some seeds. To reproduce, rerun with the printed seed. To locate the culprit, Minitest 6 ships bisection: `minitest --bisect --seed N test/` reruns subsets in subprocesses and prints the minimal combination of tests that still fails as a final command. If the test fails alone, it warns that it may not be an ordering issue. `i_suck_and_my_tests_are_order_dependent!` switches a class to alphabetical order, which hides the bug rather than fixing it. Because `srand` receives the seed, `rand` calls in tests are also repeatable for a given seed and order.

code

bash · 4 lines
bash
# the failing run printed: Run options: --seed 4242
minitest test/ --seed 4242              # replay
minitest --bisect --seed 4242 test/     # shrink to the minimal failing set
MTB_VERBOSE=2 minitest -b --seed 4242 test/

go deeper

for a junior

Recall that Minitest prints Run options: --seed N and that rerunning with --seed N replays the same order.

for a middle

Explain how the seed shuffles classes and methods, what run_order and the alphabetical opt-out do, and how --bisect narrows a failure.

for a senior

Drive an order-dependent failure to its root cause with seed replay and bisect, knowing the rake file-order caveat and when a test simply fails alone.

for a principal

Set the policy: random order stays on, quarantined classes get owners and deadlines, and CI prints seeds so any failure can be replayed.

## Why Minitest shuffles Tests that pass only in one order hide real bugs: shared state in constants, class variables, globals, caches or a database that one test leaves behind and another silently relies on. Minitest makes the order **random by default** so such coupling surfaces early instead of when someone adds a file. ## How the seed drives the order 1. At start-up Minitest chooses a seed: `--seed N` (`-s N`) if given, otherwise the `SEED` environment variable, otherwise a random value (kept below 65,535). 2. It prints it in the header: `Run options: --seed 4242`. 3. It calls `srand` with the seed and shuffles the list of test classes. 4. For each class, `runnable_methods` checks `run_order`: for `:random` (the default) or `:parallel` it re-seeds and shuffles the sorted `test_` methods; for `:alpha` or `:sorted` it keeps them sorted. The method `run_order` was called `test_order` before Minitest 6.0. A class can override it, and `i_suck_and_my_tests_are_order_dependent!` is the built-in shortcut that sets it to `:alpha`. Because Minitest seeds Ruby's global generator, `Kernel#rand` inside tests also produces the same values for the same seed and order. ## Replaying a failure - `ruby -Ilib:test test/fine_test.rb --seed 4242` - `rake test A="--seed 4242"` or `SEED=4242 rake test` - `minitest test/ --seed 4242` One caveat for rake: in Minitest 6.0.6, `Minitest::TestTask` shuffles the order in which it **requires the test files** without using the seed (the source carries a TODO about it). Method order inside a class replays exactly, but the class order can differ between two `rake test` runs with the same seed. When an exact replay matters, run the same files through the `minitest` command, which loads paths in sorted order. ## Bisecting to the culprit Minitest 6.0 absorbed the old minitest-bisect gem. From the `minitest` command: ```bash minitest --bisect --seed 4242 test/ ``` What it does: - Reruns the suite with that seed in a subprocess and checks that it still fails. - Reruns the failing test **alone**; if it still fails, warns `Tests fail by themselves. This may not be an ordering issue.` - Drops every test that ran after the failure, since it cannot be involved. - Bisects the tests that ran before the failure until it finds the smallest set that still makes it fail. - Prints a `Final reproduction:` command with just those tests. With `-n /name/` added it runs in **inverted** mode, looking for the tests that make a failing test pass. `--bisect` is recognised only by the `minitest` command, not by `ruby file.rb`, and `MTB_VERBOSE=2` shows the subprocess output when it aborts. ## Pinning order is not a fix | option | effect | when it is acceptable | |---|---|---| | `i_suck_and_my_tests_are_order_dependent!` | that class runs alphabetically | a legacy class, as a temporary quarantine | | overriding `self.run_order` to `:sorted` | same, spelled out | never as a long-term state | | fixing the leaked state | order no longer matters | always the goal | The method's name is deliberately embarrassing: it keeps the suite green while leaving the coupling in place. ## Other frameworks Ruby's other bundled xUnit framework, test-unit, defaults to **alphabetic** order within a test case and offers `--order=random` as an option, so a suite migrated from test-unit can reveal order dependence it always had.

  • With Minitest, a test fails on seed 4242 but passes with -n on its own. What does that suggest, and what next?
    It depends on something an earlier test in that order did or failed to clean up. Run `minitest --bisect --seed 4242 test/` to get the minimal set of preceding tests that still triggers the failure, then find the state they share, such as a constant, cache, `ENV` entry or database row.
  • With test-unit, how does the default test order differ from Minitest's?
    test-unit sorts tests within a test case alphabetically by default and offers `--order=random` or `--order=defined`; Minitest shuffles classes and methods with a seed by default. A suite ported from test-unit to Minitest can start failing intermittently because its order dependence was always there and alphabetical order hid it.

saying these in an interview costs you the question

  • The seed only changes test data, not the order tests run in.
  • A different seed each run means Minitest results cannot be reproduced.
  • i_suck_and_my_tests_are_order_dependent! fixes order-dependent tests.
  • ruby test/fine_test.rb --bisect works the same as the minitest command.
  • If a test fails alone, bisecting the order will find its culprit.