A case repository publishes an automated-coverage figure over its stored case definitions. What belongs in that figure's denominator, and who gets to decide?
answer
- the numerator is mechanical
- the denominator is a judgement
- in-scope, not everything in the tree
- archiving raises it with no test written
basics
~20 sThe denominator is the set of stored definitions the organisation has agreed are worth automating, not every record in the tree. Someone must own that scope in writing, or archiving, splitting and re-scoping cases will move the figure with no test written.
solid answer
~50 sAn automated-coverage figure over a case repository is a ratio of definitions claimed by automation to definitions in scope, and almost all the argument is in the second term. Left undefined, the denominator is whatever the tree currently holds, which includes retired-but-not-deleted records, exploratory charters that were never meant to be automated, data variants of one behaviour stored as separate cases, and definitions that cannot be driven at all. Each of those drags the figure down for no useful reason, and — worse — each can be removed to raise the figure without a single test being written. So the scope has to be an explicit, written rule about which definitions count, owned by a named person and reviewed like any other definition, and the figure should be published alongside the count it is a ratio of. A percentage whose denominator moves silently is not a measurement.
go deeper
Recall that an automation figure is definitions covered over definitions in scope, and that in scope is a choice somebody made rather than simply everything stored in the tree.
Explain what an undefined denominator sweeps in — retired records, exploratory charters, data variants, undrivable cases — and why each drags the figure down without telling anyone anything useful.
Show the edits that move the figure with no test written, and say what you publish alongside the ratio so a reader can tell real progress from housekeeping.
Own the governance: who writes the scope rule, why it should not be the team being measured, how exclusions are marked on the record, and the rhythm for re-examining them.
## The figure is a ratio over definitions Automated coverage in a case repository counts **records**, not code. The numerator is the set of stored definitions some automated test declares; the denominator is the set of definitions considered in scope for automation. The numerator is mechanical and falls straight out of the pairing — it is whatever the pushed results claimed. The denominator is a **judgement**, and if nobody makes it deliberately it defaults to *everything currently in the tree*, which is the worst available choice. ## What an undefined denominator drags in - **Retired-but-not-deleted definitions.** Cases for behaviour that no longer exists, kept because deleting felt risky. They will never be automated and should never have been counted. - **Exploratory charters and checklists.** Records that describe an investigation rather than a repeatable verification. Automating them is not merely hard, it is meaningless. - **Data variants of one behaviour.** The same flow stored as many definitions, one per input set, because that is how they were authored. One parameterised automated test may cover all of them while declaring one, so the denominator inflates far faster than the numerator can. - **Genuinely undrivable definitions.** Anything that needs a physical device, a third party nobody can script, or human judgement about look and feel. - **Definitions belonging to another team or product** that happen to share the tree. Each of these lowers the figure without telling you anything actionable, and a team that cannot explain the gap eventually stops trusting the number entirely. ## The dangerous property: the figure moves with no test written This is the reason the scope has to be written down rather than assumed. Consider four edits, none of which involves a line of test code: 1. **Archive a batch of stale manual definitions.** The denominator shrinks and the figure jumps. Nothing new is covered. 2. **Split one definition into two** because the automation only reached half of it. Depending on how the halves are counted, the same automation now covers one of two rather than one of one. 3. **Mark a partially automated definition as automated.** The numerator grows against work that was never done. 4. **Merge data variants into one parameterised definition.** The denominator collapses and the figure leaps, though behaviour coverage is unchanged. | Edit | Denominator | Numerator | Real coverage | |---|---|---|---| | Archive stale definitions | Falls | Same | Unchanged | | Split a definition | Rises | Same or rises | Unchanged | | Over-claim a partial case | Same | Rises | Unchanged | | Merge data variants | Falls | Falls slightly | Unchanged | Every row moves the published figure while leaving the product exactly as tested as it was. That is not an argument against the figure; it is an argument for publishing **the counts, not only the ratio**, and for treating a change in the denominator as an event that gets explained rather than absorbed. ## Who decides, and how the decision is kept honest The scope rule is a piece of policy, so it needs the things policy needs: - **A named owner.** Usually whoever owns the case tree's structure, not whoever owns the automation suite — otherwise the person measured also defines the measure. - **A written rule, not a saved filter.** *Which* definitions are in scope, stated in terms a reader can apply to a new definition on the day it is authored. A saved filter is the implementation; without the sentence behind it, nobody can tell whether a surprising exclusion is intended. - **An explicit out-of-scope marker on the record**, so exclusion is a property of the definition rather than a property of a query someone forgot to update. This also makes the excluded population reviewable, which is where the interesting rot accumulates. - **A review rhythm.** Excluded-as-undrivable ages badly: tooling improves, the third party ships a sandbox, the manual-only step gets an interface. Re-examine the exclusions periodically or the denominator quietly becomes a list of things nobody has thought about in years. ## What to publish beside the figure A single percentage compresses all of this away. Publish, at minimum, the numerator and denominator as raw counts, the size of the excluded population, and the date the scope rule last changed. Then a reader can tell the difference between *the team automated more* and *somebody archived a folder*, which is the only distinction that makes the figure worth having. When the denominator changes, say so in the same place the figure appears — an unexplained jump teaches people faster than any dashboard that the number is not to be believed.
- Why should the scope rule be owned by whoever owns the case tree rather than the automation team?Because otherwise the people measured also define the measure, and the cheapest way to improve the figure becomes editing the denominator. Separating the two does not require distrust; it just keeps a legitimate housekeeping action — archiving, merging, re-scoping — from doubling as a performance improvement nobody had to earn.
- One parameterised automated test covers twenty data-variant definitions but declares only one. What now?Either declare all twenty, if each variant is a definition you genuinely want counted, or collapse the variants into one definition with its data held as a dataset. What you must not do is leave nineteen records permanently uncovered in the denominator while the behaviour is in fact tested.
saying these in an interview costs you the question
- Treats every record in the case tree as automatable
- Lets the automation team define its own denominator
- Publishes the ratio without the underlying counts
- Raises the figure by archiving stale definitions
- Encodes scope as a saved filter with no written rule
- Never revisits which definitions were excluded as undrivable