Your GRU-versus-LSTM benchmark gap is inside seed-to-seed noise — which cell do you ship and how do you report it?

## Why this question is asked It is a judgment question, not a knowledge question. The interviewer wants to see whether you will convert a null result into a false claim, and whether you can make a defensible architecture choice when the metric you were hoping would decide it has refused to. ## Step 1 — establish that it is noise A single run of each cell tells you almost nothing. Random initialisation, data ordering and any stochastic regularisation all move the final metric, and on mid-sized sequence-tagging or acoustic benchmarks that seed-to-seed spread is routinely as large as the architectural gap you are chasing. The minimum honest protocol is several seeds per configuration, reporting the mean and the spread (a standard deviation, or the min-max range, or a confidence interval) rather than a single best number. If the two intervals overlap substantially, you have a null result. The common cheat is best-of-N: run the favoured cell five times, the baseline once, and report the best. That inflates the favoured cell by roughly the width of the seed distribution and is indistinguishable from a real improvement to anyone reading only the summary table. ## Step 2 — check the comparison was fair Two confounds account for most apparent architecture wins. **Tuning budget.** If the newer candidate got a learning-rate sweep and the incumbent got the settings someone chose a year ago, you measured tuning effort, not architecture. Give both the same search budget over the same search space, and say what that budget was. **What you held fixed.** At equal hidden size the two cells do not have equal capacity: three weight blocks against four is about 25% fewer parameters. At equal parameter count they do not have equal hidden size or equal per-step cost. Neither choice is wrong, but the write-up must say which one you fixed, because the reader's interpretation changes completely. Secondary confounds worth naming: different sequence-truncation lengths, different gradient-clipping thresholds, and evaluation on a single test split rather than several. ## Step 3 — decide on criteria the data can actually support Once accuracy is a tie, stop arguing about accuracy. Rank the criteria that are measurable and that the product actually cares about: - **Footprint and memory traffic.** The three-block cell carries about three quarters of the weights and reads about three quarters of the bytes per step. On a small on-device target — a 64-unit cell for keyword spotting on a microcontroller-class part — that is a real budget line, not a rounding error. - **Latency.** Measure it on the deployment hardware at the real batch size, which for a streaming, single-user application is one. Do not infer it from parameter counts. - **Training cost and tuning sensitivity.** Fewer gates means a slightly cheaper step and, in practice, one fewer thing to mis-initialise. - **Operational cost.** If one of the two is already deployed, monitored and understood by the team, switching to a statistically identical alternative buys nothing and costs a migration. That last point is the one candidates miss. "Keep the incumbent" is a legitimate and often correct answer to a null result. ## Step 4 — report it honestly The write-up should say, in this order: what was compared, what was held fixed, how many seeds, the mean and spread for each, the conclusion that the difference is within run-to-run variance, and the criterion that actually decided the choice. Something like: *"Across five seeds at matched hidden size, the two cells' validation scores overlap within one standard deviation; we ship the three-block cell because it is 25% smaller and meets the per-step latency budget on the target device."* This matters beyond the one decision. A team that publishes within-noise gaps as wins accumulates a folklore of architecture beliefs that no one can reproduce, and the next engineer inherits a stack of choices justified by numbers that were never real. Setting the norm — seeds reported, matched budgets, null results allowed to be null — is the part of this question that is genuinely a lead's job. ## What a strong answer avoids Do not promise that a larger benchmark or a longer run will break the tie; it may simply reproduce it. Do not pick the cell with more gates on the grounds that it is "more expressive" when your own measurements say the extra expressiveness bought nothing on this task. And do not let a within-noise result become a permanent architectural rule for the organisation.