What are the limits of code coverage as a quality signal, and how should an org set coverage policy without encouraging gaming?

## Coverage measures execution, not correctness **Code coverage** answers one narrow question: *was this code executed by some test?* It does **not** answer *did a test verify the code behaves correctly?* These differ because a test can call a method and make **no assertions** — the code runs, coverage counts it, but nothing would fail if the result were wrong. So coverage is a **necessary-but-not-sufficient** signal: low coverage definitely means untested code, but high coverage does **not** prove the tests are good. ## What coverage structurally cannot see - **Missing logic.** Coverage can only measure code that *exists*. If you forgot to handle the empty-list case and wrote no code for it, there's nothing to be uncovered — the gap is invisible. Coverage never flags a missing branch you didn't write. - **Assertion quality.** Two test suites with identical 100% coverage can have wildly different bug-catching power depending on what they assert (or don't). - **Oracle problems / integration.** Coverage of a unit says nothing about whether components compose correctly, about timing, concurrency, or real I/O. ## Goodhart's law and gaming **Goodhart's law:** *"When a measure becomes a target, it ceases to be a good measure."* Make a coverage percentage a hard gate and humans optimize the number, not the goal: - **Assertion-free tests** that execute code purely to color it green. - **Exclusion creep** — excluding packages until the bar is met. - **Trivial tests** of getters/setters/DTOs that add coverage but no value. - **Threshold inflation** demands (e.g. "go to 95%") that push teams toward the above. A bare percentage thus *can* drive worse engineering if mismanaged. ## Designing policy that resists gaming 1. **Gate on new/changed code, not the project average.** A whole-project ratio is dominated by existing code, so regressions hide and legacy gaps demand expensive retrofits. A **diff/new-code** gate (e.g. Sonar's *new code* condition, or a CI step that scopes coverage to changed lines) enforces coverage exactly where risk is introduced — high leverage, low friction. 2. **Pick the right counter.** Prefer **branch** or **instruction** over line: branch exposes untested true/false paths; instruction is formatting-independent and harder to game. 3. **Set a realistic floor, not an aspirational ceiling.** A modest, *enforced* floor (e.g. 70-80% branch on new code) beats an unenforced 95% wish. The floor's job is to stop *regressions*, not to certify quality. 4. **Constrain exclusions.** Allow only generated/untestable code; review exclusion changes like any other code so they can't be used to dodge the gate. 5. **Measure test *quality* separately with mutation testing.** A tool like **PIT** introduces small faults ("mutants" — flip `>` to `>=`, return `null`, remove a call) and reruns the tests; a mutant **"killed"** (a test fails) means the tests actually verify that behavior, a **"survived"** mutant means coverage was empty calories. Mutation score directly targets the assertion-quality gap that coverage can't see. It's more expensive to run, so apply it to critical modules or on a schedule. 6. **Risk-weight effort.** Demand high coverage where failure is costly (payments, auth, parsing) and accept less on low-risk glue. Uniform targets misallocate effort. ## How to talk about it Frame coverage as a **discovery tool** ("here's code no test touches — investigate") and a **regression guardrail** ("new code must be tested"), never as a proof of quality. Pair it with code review (do the tests assert the right things?) and mutation testing (do the assertions actually catch faults?). The mature org outcome: coverage finds the gaps, review and mutation testing ensure the tests filling them are real — and no one is rewarded for a green number alone.