Under `go test -shuffle=on` a package's tests fail, though each passes alone. How do you find the cause?

## Why the failure exists at all `go test` compiles one test binary per package and runs all of that package's tests inside a single process. Package-level variables are therefore shared by every test in the package, initialized once before the first test function runs, and never reset. Tests in *different* packages get different processes, which is why this class of failure never crosses a package boundary — and why "it passes when I run just that file" is not evidence of anything. The default execution order is source order, which is stable enough that a suite can carry a hidden dependency for months. `-shuffle=on` randomizes the order of top-level tests and benchmarks, exposing it. Because the seed is random per run, the failure looks intermittent: green on your machine, red in the job that happened to draw a bad order. ## Making it reproducible first Do not start reading code. Get a deterministic reproduction: 1. Run with `-shuffle=on` and capture the seed it prints. 2. Re-run with that seed. The same order gives the same failure. 3. Add `-count=1`. Without it, a cached successful result can be replayed and you will chase a ghost. `-count=1` is the standard way to force the test to actually run. 4. If the failure still moves, suspect concurrency rather than ordering, and run the same command under the race detector. ## Narrowing it to a pair Order dependence is almost always a relationship between two tests: one leaves state behind, another reads it. Use `-run` with a regular expression to run subsets — `-run 'TestA|TestB'` — and bisect the failing order until you have the smallest set that still fails. Two names is usually the answer, and it is often obvious which of the two is the writer once you see them together. A useful shortcut: run the suspected victim immediately after each other test in turn. Whichever predecessor makes it fail is the writer. ## What the writer usually is In practice the shared state is one of a small set of things: - A package-level configuration or client variable that a test swaps for a fake. - A clock: a package-level `var now = time.Now` that a test replaces with a fixed function. - A global registry map that a test adds an entry to. - A handler registered on a shared default multiplexer, which a second registration of the same path will panic on. - Process-wide state that is not yours: the environment, the working directory, a temporary file at a fixed path. All of them have the same shape: something mutable that outlives a single test. ## Restoring correctly, if you must touch a global When a global genuinely has to be swapped, restore it with `t.Cleanup` rather than `defer`. Two reasons. `t.Cleanup` still runs when the test ends early through `t.Fatal`, which unwinds via `runtime.Goexit` — a `defer` in the test function does run in that case, but a `defer` in a helper the test called does not help you once the helper has returned. More importantly, a test that calls `t.Parallel` is paused and resumed only after its parent's function body has returned, so a `defer` in the parent restores the value *before* the parallel children run, while a `t.Cleanup` registered on the parent runs after they have all finished. For environment variables specifically, `t.Setenv` sets and restores for you, and it deliberately refuses to be used from a parallel test — because there is one environment per process and no way to isolate it. Be aware that `t.Parallel` turns order dependence into a genuine data race: two tests mutating the same variable at the same time. Running the suite under the race detector will find that, and it is worth doing on this class of bug even though the ordering failure itself is not a race. ## The structural fix Restoring state is damage control. The fix that ends the category is to stop sharing: make the thing a field on a struct that each test constructs, pass the clock in as a parameter, build a registry value per test, use a fresh multiplexer instead of the shared default one, and let each test own a temporary directory. Once no test can observe another test's writes, order stops mattering and tests become safe to parallelize — which is the real payoff, since a suite that can run in parallel is usually several times faster. ## Keeping it from coming back Turn shuffling on in CI so a new ordering dependency fails on the pull request that introduces it, not months later. Keep the printed seed in the job output so a red build is reproducible. And treat "this test needs to run first" in a review the way you would treat a global variable, because that is what it is. ## What to tell the person triaging If the job is green locally and red in CI, the first question is what differs: a random shuffle seed, a cold test cache, a different value of `GOMAXPROCS`, or parallelism settings. Order dependence explains all four, and the reproduction recipe above turns an intermittent red into a deterministic one within a few minutes.