What must a quarantine policy for unreliable tests contain so that quarantine does not become permanent?

### What quarantine is, and what it is not **Quarantine** moves a case out of the set whose result blocks the pipeline, while *still executing it and still recording its results*. That second half is what separates quarantine from deletion. A quarantined case keeps producing history — you can see whether it is still disagreeing with itself, whether it has started failing consistently (which means it has found something real), and how long it has been out. Quarantine exists because of a compounding problem: one case that disagrees with itself poisons the verdict for everyone, and a suite that cries wolf is worse than no suite, because it trains people to re-run rather than read. Taking the offender out restores a believable signal for the other several thousand cases. That is a genuine, defensible engineering move. But the cost is precise and it is usually understated: **quarantining is deleting the coverage of a real behaviour, silently, with no record in the risk register unless you make one.** Whatever that case asserted is now unguarded, and the exclusion is invisible because the pipeline is green. ### The policy, clause by clause A quarantine policy that does not become a graveyard has, at minimum, six clauses. 1. **An evidence-based entry criterion.** Not "someone was annoyed". Something like: the case produced conflicting outcomes on the same revision at least twice within a rolling two-week window, or it spoiled more than a stated number of runs. The criterion is checked against the results history, not asserted in a message. 2. **A named owner assigned at entry.** A team name is the weakest acceptable answer and an individual is better. Quarantine with no owner is deletion with extra steps. 3. **A timebox with a default action on expiry.** Two weeks is a common choice. The crucial part is that the default when the clock runs out is *not* renewal — it is either repair or deliberate removal with an explicit statement of what is no longer checked. Renewal must require the same effort as the original entry. 4. **A hard cap on the list.** Give the list a fixed number of slots — say twelve. Entering quarantine when the list is full requires evicting something, which forces triage exactly when the team is most tempted to skip it. A budget converts an unbounded backlog into a queue. 5. **Continued execution and reporting.** Keep running them, off the blocking path, and publish the list with per-entry age. A quarantined case that begins failing *consistently* has stopped being unreliable and has started being a defect report. 6. **Visible risk.** Record, in one line per entry, the behaviour that is now unguarded. This is what makes the cost legible to someone deciding whether to accept the entry. Publish two derived numbers next to the flake-rate trend: **list size** and **age of the oldest entry**. Without them, a falling flake rate is unreadable, because the cheapest way to improve reliability numbers is to stop counting cases. ### The incident that makes the cost concrete A ride-hailing dispatcher team quarantined a case asserting that a support-desk role could not reassign a ride belonging to another operator. It had disagreed with itself three times in a fortnight — the environment it needed was genuinely slow to prepare — and taking it out restored a believable 27-minute suite. There was no timebox and no owner. Nine weeks later a refactor of the role-resolution path shipped a **permission escalation**: support-desk accounts could reassign any ride in the region. The quarantined case failed correctly and consistently throughout, in a report nobody read, and the escalation was found 12 days after release by an operations lead reading an audit trail. Two lessons, and an interviewer wants both. First, the missing clauses were the timebox and the owner, not the decision to quarantine — the decision was reasonable given a suite nobody could believe. Second, the quarantined case *was still producing the answer*; the failure was that no one was obliged to look. A policy that re-reads its quarantine results weekly would have caught it in days rather than weeks. ### Choosing what may be quarantined at all Not every case should be eligible. A sensible policy carves out an ineligible set — the cases guarding authorisation boundaries, money movement, data deletion and similar irreversible behaviour. For those, an unreliable case is an incident of its own: you fix the case or the environment, or you accept a slow suite, but you do not quietly stop asking whether a support role can take over somebody else's ride. Deciding that carve-out in advance, in calm conditions, is far easier than arguing it at 18:00 with a release waiting. ### Alternatives worth naming Quarantine is one option among several, and a strong answer positions it: - **Fix it now** — right when the offender is one case and the cause is fresh. - **Move it to a slower tier** where its unreliability blocks nobody but the signal is retained on a schedule. - **Make it non-blocking but loudly reported** — a middle ground where the case still shows red in a report the team reviews. - **Delete it deliberately**, recording what is no longer checked. Honest deletion is better than a permanent quarantine that pretends the coverage still exists. - **Fix the environment rather than the case**, when several unrelated cases became unreliable at once — that pattern points at shared infrastructure, and quarantining them one by one treats the symptom. The judgement being assessed is whether you understand quarantine as a *loan against the safety net*: useful, occasionally necessary, and dangerous exactly in proportion to how long it stays outstanding and how invisible the interest is.