How do you decide which tests may run on an engine substitute and which must use the real engine?

## The decision, stated properly The question is not "substitute or real engine" but "which claim does each case make, and which engine can support that claim". Framed that way the split is mostly mechanical. **Cases a substitute may serve.** Their subject is application behaviour and the store is incidental: branching, validation, calculation, mapping, orchestration between components, error handling. Swapping the engine underneath such a case does not change what it proves. **Cases that require the real engine.** Their assertion *is* engine behaviour: a query with non-trivial filtering, grouping or ordering; anything relying on an index; uniqueness, cascade, check and deferred-constraint behaviour; collation and case sensitivity; numeric and fractional-second precision; isolation, locking and deadlock handling; and the migration history. Add one pragmatic rule that catches the rest: any statement using an engine-specific feature is not permitted on the substitute, because the substitute's version of that feature is a reimplementation. ## Make it a tier, not a preference If the choice is left to each author, the codebase ends up with both engines used arbitrarily and nobody able to say what the suite proves. Encode it instead: - one **fast tier** on the substitute, running on every push, with a stated budget — for example, the tier must finish inside its 92nd-percentile target of seven minutes forty seconds, and a change that pushes it past that must buy the time back; - one **engine tier** on the real engine, running before merge or on a short schedule, deliberately smaller and deliberately not a mirror of the fast tier; - a placement rule in the contribution guide that an author can apply without asking, and a review prompt when a case in the fast tier touches a query. The budget matters more than it looks. Almost every argument for keeping a substitute is really an argument about feedback latency, and once the number is written down the argument becomes measurable rather than a matter of taste. ## The check that keeps the substitute honest A substitute is a claim: *behaviour relevant to these cases is the same on both engines*. Nothing in a green fast tier tests that claim. The check that does is a **replay** — run the same cases against both engines and compare. Any divergence is a finding: either the production engine behaves in a way the code did not anticipate, or the substitute has drifted and some case is no longer trustworthy on it. Run the replay on a schedule rather than per commit, because its value is trend, not gating. Record what it finds; a replay that has produced nothing for months is either evidence the substitute is fine or evidence the cases in it never touched the engine. Both are useful conclusions and they are distinguishable by looking at what the replayed cases actually assert. ## Costs to hold in view - **Two schemas.** If the substitute's schema comes from anywhere other than the production migration path, drift is guaranteed and silent. - **Reverse pressure on production code.** The failure mode to forbid outright: dropping an index, rewriting a set-based statement as a loop, or avoiding a feature so the substitute still passes. At that point the test engine is designing the system, and the cost is paid forever in production performance. - **Two dialects in test setup.** Setup fixtures written for one engine and patched for the other are an ongoing tax and a source of cases that pass for the wrong reason. - **False confidence.** The most expensive cost, and the hardest to see, because its symptom is an absence of failures. ## The worked case A subscription renewal job is the most engine-dependent flow in a system: it selects a due set at a boundary, updates rows under concurrency, and depends on an index to be viable at all. Placing its cases in the fast tier would be the classic error — every one of its assertions is about the engine. In one such split, the renewal boundary query was left on the substitute, and a comparison that behaved as a strict inequality on the production engine's truncated timestamps produced an off-by-one at the boundary: 118 subscriptions renewed on two consecutive runs. The engine tier had no case for the boundary because the boundary case already existed and was green. The lesson is about placement, not about the assertion. ## Revisiting the decision Treat it as reversible and set the evidence that would reverse it: escaped defects attributable to the gap over a period, the measured wall-clock cost of moving a tier onto the real engine, and how much of the fast tier genuinely touches the store at all. Often the third number is the surprise — a large share of substitute-backed cases turn out not to need a store, and moving them off it shrinks the problem instead of solving it. Be honest that the wider question is contested. One camp argues substitutes are legacy now that a disposable real engine is routine; another points to constrained build agents, licensing, start-up cost and offline work. The defensible position is not a slogan but the split above plus a replay that keeps checking whether the split still holds.