skip to content

Which oracles can a tester use when no written specification exists?

level: middleimportance: should knowfreq 54%

answer

  1. No document does not mean no basis
  2. Ask what it should agree with
  3. History, claims, peers, itself, purpose
  4. External rules beat internal taste
  5. Two weak agreeing oracles make a report

basics

~20 s

Consistency oracles: with history, with the product's own claims, with comparable products, with itself across features, with users' expectations, with its stated purpose, and with applicable standards - plus whether the behaviour is explainable at all.

solid answer

~40 s

Absent a specification you fall back on consistency: the behaviour should be consistent with **history** (how the previous release behaved), with the product's **claims** (help text, labels, release notes), with **comparable products** solving the same problem, with **itself** - the same fact in two screens must agree and sibling features must behave alike, with **users' expectations**, with the feature's **stated purpose**, and with any **standard or statute** that applies. Two more are worth naming: **familiarity**, meaning it resembles a failure pattern you have seen before, and **explainability**, meaning nobody can give a coherent account of why it behaves this way. A widely taught mnemonic bundles these consistency headings so a tester can walk the list under pressure. Each is weak alone; two agreeing is a report, and you always say which one you used.

go deeper

for a junior

Recall that documents are not the only basis for a verdict, and be able to name three consistency oracles - the previous behaviour, the product's own labels and help text, and the product agreeing with itself.

for a middle

Walk the consistency list under pressure and give a concrete failure mode for each one you name. Interviewers here want to hear that you stack weak oracles and state which one a finding rests on.

for a senior

Show judgement about ranking: reach for external standards in regulated areas, distrust history when the change was intentional, and explain why two oracles from the same source agreeing proves nothing.

for a principal

Own the limit of the whole family - a product can be uniformly self-consistent and uniformly wrong - and decide where the organisation must obtain an independent external oracle rather than keep stacking cheap internal ones.

## The situation 'There is no specification' is the normal case, not the exception. Even where documents exist they cover a fraction of the behaviour, go stale, and say nothing about the interactions that break. So the practical skill is knowing what else can render a verdict - and consistency is the answer, because a system that contradicts something it ought to agree with is showing you a problem even when nobody wrote down what the right answer is. ## The consistency headings **History.** Does it behave as this product behaved before? A change nobody asked for is at minimum a question. Weakness: it treats the previous behaviour as correct, so it is blind to defects that have always been there, and it produces noise whenever the change was intentional. **Image.** Does it fit the standard of quality the organisation presents to the world? A truncated total on a customer-facing document is a problem for a firm that sells accuracy, even if it violates no stated rule. **Comparable products.** Does something else that solves the same problem behave this way? Strongly suggestive and never binding - the other product is entitled to a different policy, and it may be wrong too. Also worth including: comparable *parts of the same product*, and the product's own competitors in adjacent markets. **Claims.** Does it do what the help text, the labels, the release notes and the marketing say it does? Very cheap, very effective, and a contradiction is difficult to dismiss because someone in the organisation wrote the claim. **Users' desires.** Would a reasonable person doing this job want this behaviour? This is a genuine oracle and also the easiest to overreach with - it is your model of the user, not the user. Establishing what users actually want is a research discipline of its own. **Product - internal consistency.** The same fact presented in two places must agree, and features in the same family should behave alike. This is the strongest oracle available with no documents at all, because a contradiction is *self-evidently* a defect: both views cannot be right. What it does not tell you is which side is wrong. **Purpose.** Does it serve what the feature exists for, whether or not it matches any written rule? A report that is technically accurate but unusable for the decision it was built to support fails a purpose oracle. **Standards and statutes.** Rounding conventions, date and currency formats, accessibility rules, tax and employment law. These are external and independent, which makes them the strongest oracles available in a regulated area - and worth hunting for before you argue from taste. Two late additions to the same family are commonly taught alongside them: **familiarity** - it resembles a failure mode you have seen before, so it deserves a look - and **explainability** - nobody, including the developer, can give a coherent account of why it does this, which is itself a signal. ## Using them well A worked example. A payroll engine has no requirement document for its rounding behaviour. The payslip shows a net figure; the ledger export for the same period shows a total that is off by a fraction of a currency unit; the year-to-date summary shows a third figure. That is a pure internal-consistency finding: three views of one number, mutually contradictory, and a defect is proven without any specification at all. The next move is the standards oracle - the jurisdiction's published rounding rule - which is external and independent, and which tells you *which* of the three is wrong. History would have been a poor choice here, because the previous release is exactly what is being replaced. Three habits separate a middle answer from a good one: 1. **Stack them.** One weak oracle is a hunch; two independent weak oracles pointing the same way is a report worth writing. 2. **Name the one you used.** 'This contradicts the on-screen label' and 'this differs from the previous release' land very differently with a developer, and both land better than 'this looks wrong'. 3. **Prefer independence.** Where two oracles derive from the same source - a claim written from the same misunderstanding as the code - agreement between them proves nothing. ## Where it stops Consistency oracles let you find problems without documents; they do not let you declare correctness. A system can be perfectly self-consistent, consistent with its own claims, consistent with its history and uniformly wrong. Knowing that is what keeps a tester hunting for an independent source rather than settling for the cheapest available agreement.

  • Which consistency oracle is strongest when nothing external is available, and why?
    Internal consistency. If the same quantity appears in two places and the values differ, a defect is proven without any external reference, because both cannot be right. Its limit is that it identifies the contradiction, not the culprit - you still need an independent source, such as a published rule, to say which side is wrong.
  • What is the risk of leaning on a comparable product as your oracle?
    It is suggestive, never binding. The other product may have made a different but equally valid choice, may be optimising for a different user, or may simply be wrong. Use it to raise a question and to argue about user expectation, not to assert a requirement. A finding that rests only on 'the other one does it differently' is easy to reject.
  • When would you deliberately not use the previous release as an oracle?
    When the previous release is the thing being replaced or is already suspected, when the change under test is an intentional behaviour change, or when the defect class you are hunting is long-standing. In all three the history oracle either enshrines the old error or fires constantly on intended differences, so an external rule is the better basis.

saying these in an interview costs you the question

  • Says no testing is possible without a specification
  • Treats a comparable product's behaviour as a requirement
  • Uses only personal taste and calls it user expectation
  • Ignores help text and labels as a source of claims
  • Assumes self-consistency means the behaviour is correct
  • Never names which oracle a finding contradicts

context