How do you estimate whether an automated case will repay the cost of writing and maintaining it?

### Why a number beats an opinion "We should automate this" and "that is not worth automating" are equally unfalsifiable. A rough model turns the argument into one about inputs, which is a much better argument to have: someone can dispute your estimate of upkeep, and disputing it produces information. ### The model For a single candidate case, over a chosen horizon: ``` cost = authoring + (upkeep_per_period * periods) + (triage_per_failure * expected_failures) return = (manual_minutes_per_run * runs_over_horizon) + value_of_earlier_detection ``` And the break-even run count, ignoring upkeep for a moment: ``` break_even_runs = authoring / (manual_minutes_per_run - automated_triage_per_run) ``` Four things about this model matter more than its arithmetic. **The denominator is a difference, not the manual cost.** An automated run is not free of human time: someone reads the result when it goes red. If a case fails spuriously often, the triage term can eat most of the saving and push break-even out beyond the horizon. **Upkeep is a recurring subscription.** It is the term candidates leave out, and it is the one that decides most real cases. Upkeep scales with how often the behaviour, its data or its interface changes, not with how often the case runs. A case guarding a rule that changes twice a quarter is a standing bill; a case guarding a rule untouched for two years is nearly free after authoring. **Runs over the horizon, not runs in principle.** The horizon ends when the behaviour is replaced, the feature is retired, or the interface is rebuilt — whichever comes first. A case that repays after 180 runs, on a behaviour that will be redesigned after 60, loses money however elegant it is. **Earlier detection is real but not in the arithmetic.** Catching a permission regression in eight minutes rather than during a manual pass three weeks later is worth something, and for severe classes of defect it is worth more than the whole labour saving. Add it as an explicit risk term rather than pretending the labour arithmetic alone justifies the case. Be careful with the numbers you attach: the popular multipliers for how much more a late defect costs are contested and their original basis is disputed, so cite them as an argument for direction, not as a measured constant. ### A worked example A 4-person team on a marketplace bidding engine weighs one candidate: a check that an account without the seller role cannot cancel a live auction — the area that produced a permission-escalation defect two releases ago. - Authoring, including seeding two accounts and a live auction and getting it running in the pipeline: **3 h 40 m**. - Manual cost of one honest run, including setup: **11 minutes**, and the team admits that in practice it gets skipped under time pressure. - Triage per automated run, averaged over green and red: **0.4 minutes**. - Expected upkeep: the rule has been stable 21 months, so **about 25 minutes per quarter**. - Runs over the horizon: the pipeline runs it on every change, roughly **190 times** in the first year. Break-even on labour alone: 220 minutes / (11 - 0.4) is about **21 runs** — reached in the first fortnight. Upkeep of 25 minutes a quarter is trivial against a saving of about 10.6 minutes per avoided manual run. Add the severity of the regression it guards, and it is not a close call. Now the same team's second candidate: a check on the redesigned bid-confirmation screen, authoring 2 h 10 m, manual cost 3 minutes, running perhaps 9 times before the promotion ends, on a screen being rebuilt. Break-even is around 43 runs against a horizon of 9. The arithmetic says no, and now the team has something better than a taste argument to say so. ### Arguing against automating everything The strongest version of the argument is arithmetic, not principle. Suppose a 4-person team can sustain, say, six hours a week of automation upkeep in total. Every case added draws on that fixed budget forever. Once the budget is exceeded, the marginal case is not free — it is paid for out of the repair time of existing cases, which then start failing for reasons nobody chases. That is how a set of cases stops being trusted. So the real constraint on "automate everything" is not authoring capacity but *upkeep capacity*, and it is the number to put in front of whoever set the percentage target. ### Presenting the estimate Give a range, name your assumptions, and say which input the answer is most sensitive to — usually upkeep or the horizon. An estimate that says "this repays in about three weeks unless the interface is rebuilt before then, in which case never" is far more useful to a decision-maker than a single confident number.