A test asserting a serialized map's key order passes locally but fails in CI — how do you fix it?

## First, reclassify the ticket "Flaky test" invites a retry. This is not flaky: within each environment the behaviour is almost certainly deterministic and reproducible. What differs is an *input to the layout* that nobody thought of as an input. Getting the classification right is most of the senior signal in this question, because it decides whether the team adds `retry(3)` or removes an incorrect assumption. ## Enumerate what could differ between the two environments - **Data volume.** The local fixture may hold fewer entries than the one CI builds, or CI may run the suite in an order that leaves extra entries in a shared fixture. If the table crosses its load-factor threshold in one environment and not the other, capacity differs, every key's slot differs, and the sweep order differs. - **Runtime version.** Laptops drift from the pinned build image. A change in growth policy, in the reduction from hash to index, in which end of a chain new entries join, or in a bucket's overflow representation moves the order without any change in your code. - **Per-process variation.** Some implementations deliberately vary where iteration starts or mix a per-process value into hashing, so two runs of the same binary on the same data disagree. When that is the case, the test fails locally too — just not every time, which is the most confusing variant of this bug. - **Key content.** If keys embed anything environment-dependent — a path, a hostname, a generated identifier — the hashes themselves differ, and so does the layout. A quick discriminator: run the local suite with the CI fixture size, and run it twice in one process versus twice in fresh processes. Same-process repeatability with cross-process disagreement points at per-process variation; disagreement that appears only above a certain row count points at a resize. ## The fix menu, with the tradeoffs stated **1. Make the consumer order-insensitive.** Parse the serialized document and compare the resulting structures, or compare sets of lines. Cheapest, and it fixes the actual error: the test was asserting on something the system does not specify. Downside: you lose the ability to catch a genuine ordering regression, which matters only if order is genuinely part of the contract — and if it is, the producer must guarantee it, which is fix 2. **2. Make the producer deterministic — sort keys at the output boundary.** Now the artifact has a stated order, the test can assert on exact text, and any human diffing two archived versions sees only real changes. Costs an `O(n log n)` sort per emission and, importantly, changes the artifact for existing consumers — a one-time coordinated change. This is usually the right answer when the output is archived, diffed, signed, or compared across runs. **3. Hold the data in an insertion-order-preserving map.** Correct when arrival sequence carries meaning — an event log, a form's declared field order, anything where sorted output would be wrong. Costs extra memory per entry and a little write-time bookkeeping, and it only helps if the insertion sequence is itself deterministic; feeding it from another unordered iteration just moves the nondeterminism upstream. **4. Declare the order irrelevant and fix the specification.** Sometimes the honest outcome is that neither producer nor consumer should care, and the assertion existed only because it was easy to write. Document that the order is unspecified and delete the assertion. ## The fixes to refuse - **Pinning or aligning the environment** so both sides agree today. The assumption survives, untested, until an upgrade or a data-volume change reactivates it — in production rather than in CI, where it was helpfully caught. - **Retrying the test.** Same objection, plus it trains the team to ignore a real signal. - **Sorting inside the test only.** This makes the suite green while leaving the artifact nondeterministic, which is the worst combination: the producer is still wrong and you have removed the alarm. ## Preventing recurrence Make the class of bug visible rather than relying on memory: normalize golden files by sorting at write time so the property is enforced by the producer; prefer structural comparison helpers in the test toolkit over raw-string equality for generated documents; and add a review habit — any map iteration that feeds a serialized artifact, a signature, a hash of output, or a user-visible list needs an explicit ordering decision at that point. The failing CI run is a gift: it caught a latent assumption before an auditor did.