Which checks do you deliberately leave manual instead of automating, and why?

### "Not automated" is a decision, not a failure Interviewers ask this question because the reflex answer — "automate everything" — is expensive and common. A strong answer names the families of check that stay manual and gives the reason for each, because the reason is what generalises to a case the interviewer has not mentioned. ### One-offs and near-one-offs Authoring cost is amortised across runs. A check that runs once has nothing to amortise against. The classic instance is a data migration verified the day it happens: careful, high-stakes, and never repeated in that form. Near-one-offs are the trap — a check expected to run "a few times" during an eleven-day promotion looks repetitive but will not accumulate enough runs before the promotion ends. The test is the count over the *horizon*, not the presence of repetition in principle. The exception worth naming: a throwaway harness written to *find* a defect rather than to guard against one. Driving several hundred generated inputs at a rule to hunt for a bad case is automation whose payback lands inside a single sitting, and it is fine to delete afterwards. That is a different economic decision from adding a case to a set that runs forever. ### Subjective judgements Some outcomes are only meaningful to a person: whether copy reassures a nervous buyer, whether a layout is pleasant, whether an error message helps rather than blames, whether the flow feels fast. There is no oracle a machine can evaluate here — the best it can do is detect that pixels changed, which is not the same claim and generates failures for every intentional redesign. Usability and aesthetic judgement stay with people, and pushing them into automation produces cases that either assert nothing or assert the wrong thing. ### A poor or missing oracle An oracle is the thing that says the behaviour is wrong. It can fail in two ways. First, nobody can state the expected outcome: the rule is undocumented, contested between two people, or genuinely "whatever the system does today". Encoding today's behaviour then locks in whatever defect is currently present, and every future correction shows up as a red case. Second, the outcome legitimately varies — an ordering that is not specified, a value that depends on the moment, a recommendation that is allowed to differ. A case built over a varying outcome either asserts something so loose it can never fail or fails on unchanged behaviour. Get the oracle agreed first; then decide about automating. ### Unstable interfaces When the surface a case drives is mid-redesign, the case is a liability: it will be rewritten before it has run enough times to repay. The right move is to wait for the surface to settle, or — where possible — to drive the same rule through a more stable channel and check it there. "Wait" is a real answer, not a cop-out, provided somebody has said when to look again. ### A worked example A 4-person team on a marketplace bidding engine sorted a backlog of 47 candidate checks and left 12 manual. Two were launch-day migration verifications. Five were judgements about the redesigned bid-confirmation screen — clarity of the countdown, tone of the outbid message. Three were checks over a recommendation ranking whose expected order nobody would commit to. Two drove a seller console being rebuilt over the next quarter. Against those, the team automated the permission rules around cancelling an auction immediately: exact oracle, stable rule for 21 months, and a permission-escalation defect in that area two releases earlier. Notice what the split produced. It did not produce twelve untested behaviours — it produced twelve behaviours checked by a person, deliberately, with the reason recorded next to each. The record matters, because the reasons expire: an interface settles, a promotion becomes permanent, an oracle finally gets agreed. A list of manual-for-now checks with no revisit date silently becomes a list of things nobody looks at. ### Automate the arrangement, keep the judgement Even inside a manual check, most of the minutes are mechanical: creating accounts with the right roles, seeding auctions in the right state, moving the clock, capturing evidence. Automating that arrangement can turn a 14-minute manual check into a 2-minute one and reduces the human error rate of the setup. The judgement stays with a person; the tedium does not. Candidates who answer only "automated or manual" miss this, and it is often the highest-return move available.