Your CI pipeline retries each failed browser end-to-end test up to twice and reports the run green if a retry passes. What does that policy buy you, what does it cost, and what would you put around it?

## What a retry actually is Automatic retry says: this test's result is a sample from a distribution, and we will take up to three samples and keep the best. That is a defensible engineering choice for a browser suite, where a runner can lose a container, a network can blip, and a shared environment can stall. It is not a defensible answer to "why is this test nondeterministic", and confusing the two is the failure mode the question is probing. ## What it buys Throughput and morale. Without retries, a suite with a 0.5% per-test flake rate and 400 tests fails most runs by arithmetic alone, and the team learns that red means nothing — which is a far worse outcome than retries, because it destroys the gate's meaning through habit rather than policy. Retries also let you keep a large end-to-end suite on the merge path at all, and they buy time to deflake without stopping delivery. ## What it costs **Signal.** A passed-on-retry rolled into a green run erases the only evidence that anything was wrong. The flake rate can double over a quarter with no visible symptom until it crosses the point where three attempts are not enough. **Real bugs.** A double-submit that fires one time in twenty, a race between two requests, a stale-cache flash — these fail intermittently in CI exactly as flakes do. Retrying them ships them. This is the strongest argument for treating passed-on-retry as an event that must be looked at rather than absorbed. **Money and time.** Retrying a slow browser test three times triples its wall clock on the runs that need it, and a badly flaky suite spends a large fraction of its compute re-running. **Culture.** Once "just re-run it" is the accepted first response to red, it becomes the response to genuine failures too. ## The guardrails - **Make the flake visible.** Report passed-on-retry as its own outcome, per test, into a durable store. The number you manage is the flake rate, not the individual failures. - **Gate the trend, not the test.** Individual flaky tests do not block merges; a suite whose flake rate exceeds its budget does. That inverts the incentive: the team is accountable for the aggregate, and deflaking becomes a scheduled cost rather than heroism. - **Cap the count.** One or two retries. A test that needs three attempts is not flaky, it is broken, and a high cap mostly buys compute. - **Retry only what is safe to retry.** A test that creates data, sends an email, or moves an external resource may fail differently on the second attempt because the first left residue. Retrying such a test produces confusing failures; the precondition for retry is that the test can start from a clean state. - **Separate infra retries from test retries.** Losing a runner is an infrastructure event and should be retried at the job level without counting against the flake budget. A test that failed on its own merits is a different signal, and merging the two hides both. - **Turn retries off where truth matters.** Release verification, smoke tests against production, and any test guarding a payment or auth path should report the raw result. - **Quarantine with an expiry.** Chronically flaky tests come off the gate with a named owner and a date, keep running, and report. An expired quarantine entry should itself fail the build so the decision — fix it or delete it — is forced rather than deferred. ## The organisational question underneath The retry policy is really a question about who owns suite reliability. If nobody does, retries are a slow leak: each team adds one more flaky test, nobody's dashboard turns red, and in a year the suite is advisory. Making the flake rate a visible, owned, budgeted number — with the retry mechanism feeding it rather than hiding it — is what keeps an end-to-end suite worth running. The honest framing in an interview is that you keep retries *and* you make them expensive to rely on.