How would you decide which parts of a regression pack should become property-based tests?

### The decision is per behaviour, not per team The failure mode I most want to avoid is a mandate - *every module gets a property* - which produces tautologies in the modules where no rule exists and a maintenance burden nobody defends. Property-based tests and example tests supply different oracles, and the question for each behaviour is which oracle is available and what it costs. ### Where properties earn their place Four signals, and I want at least three of them: - **The input space is large and structured**, so no realistic set of hand-written cases covers it. Anything parsing, encoding, merging, ordering, deduplicating or normalising qualifies. - **A rule exists that is independent of the implementation** - an inverse to round-trip against, an invariant from the requirement, a naive reference implementation, or a metamorphic relation. Without independence the property is a tautology. - **Defects are input-shaped**: boundaries, duplicates, equal keys, empty collections, unusual character classes, arithmetic edges. These are exactly what a human writing examples fails to imagine. - **A single case is cheap**, milliseconds and in-process, so thousands of runs are affordable. In the playlist service, the share-string codec, the queue builder with its cap and pinning rules, and the batch-update conflict resolution all score four out of four. The conflict resolution is where it paid: at a 1,200-request-per-minute peak, two updates to one playlist within a window are routine, and the defect was an **ordering assumption** no example test author had thought to write down. ### Where example tests still win **The expected value is a decision, not a rule.** Rounding tables, tier boundaries negotiated with finance, the exact wording of a notification, a regulated figure. Any *property* over these would restate the table, and a test that restates the table is a copy of the specification with the same defects. Write the examples, one per documented case. **The test is a communication artefact.** An acceptance case a product owner or an auditor reads is a scenario with concrete values. Its readability is the point; replacing it with a universally quantified rule loses the audience that the artefact exists for. **Each run is expensive.** End-to-end journeys, tests touching real infrastructure, anything measured in seconds. Property-based testing trades many cheap runs for coverage of the space; at forty seconds a run that trade collapses. Push the property down to the pure core and leave the expensive layer with a handful of examples. **The failure must be deterministic and legible in a gate.** A release gate that can fail on an input nobody has seen creates a triage decision at the moment a team is least able to make it, and the pressure to disable it is enormous. Determinism is a property of the artefact, not of its quality. **Regression pinning.** A specific past incident deserves a specific named test, permanently, independent of whether a generator would rediscover it. ### The shape I default to Properties run in a generated-input job outside the release gate - on merge to the mainline and nightly, with a larger budget nightly. Findings are shrunk, triaged and pinned as example tests inside the gate. The gate stays deterministic and fast; the exploration keeps running with a bigger budget and a longer time horizon. Where a property is cheap, mature and has not produced a false alarm in months, it graduates into the gate with a small fixed run count. ### The organisational costs to price in A good property is harder to write than a good example, and there are two ways to write a bad one: the **tautology** that can never fail, and the **over-strong** clause that fails on legitimate input. The second is more damaging, because a suite whose failures are not trusted is worse than no suite. So review properties for **provenance** - each clause traceable to a requirement, with a plausible defect it would catch - and count the triage cost of generated failures as part of the cost of ownership. Team capability matters too: if two people can read a property and disagree about what it claims, it is not ready to gate anything. ### What I do not decide on I would not justify the split with a claimed defect-detection rate. The comparative evidence between testing styles is genuinely contested and heavily context-dependent, and industry-wide numbers of that kind do not survive contact with a specific codebase. What I would do is measure locally: over two or three quarters, record which style caught each escaped defect and what each style cost in triage time. That is a number about this system, and it is the only kind worth steering by.