skip to content

How does a coverage tool record which lines and branches actually executed?

level: middleimportance: should knowfreq 48%

answer

  1. Something is added to the code first
  2. Probes sit at block boundaries
  3. Rewrite the source, or the compiled artefact
  4. Hits dumped, then mapped back to lines
  5. A process killed mid-run flushes nothing

basics

~20 s

A coverage tool inserts probes at the entry of each basic block, rewriting either the source before compilation or the compiled artefact at build or load time. Each run dumps the probe hits, which a report step maps back to source lines.

solid answer

~50 s

Coverage is measured, not inferred: the tool adds probes — counters or flags — at the entry of every basic block, so a hit proves that block ran. The rewriting happens in one of three places: on the source before compilation, on the compiled artefact offline at build time, or on units as they are loaded, by an agent attached at start-up. During the run the probes update an in-memory table; on exit, or on an explicit dump, that table is written as execution data. A separate report step joins the probe table against line-number information kept in the artefact to produce per-line and per-branch results, and can merge data from several runs. The consequences are practical: data buffered by a process killed mid-run is lost, and units loaded before the agent attached are never instrumented.

code

pseudocode · 10 lines
pseudocode
function acceptVisitForm(form):
    probes[17] = probes[17] + 1
    if isValid(form):
        probes[18] = probes[18] + 1
        return store(form)
    probes[19] = probes[19] + 1
    return reject(form)

onProcessExit():
    writeExecutionData(probes)

go deeper

for a junior

Know that coverage comes from an actual run, not from reading the code, and that the tool adds counters to the code to see what executed. Being able to say "the run writes a data file, a report step turns it into percentages" is enough here.

for a middle

Explain probes at basic-block entries, the three rewriting points — source, compiled artefact at build time, or units as they load — and how a report joins probe hits to lines using retained line information. Mention merging several runs.

for a senior

Diagnose a suspicious report: a killed process that never dumped, an agent attached after the code was loaded, an un-instrumented service in a second process, generated code inflating the denominator. Say which evidence separates these from a real drop.

for a principal

Own where instrumentation sits in the pipeline and what it costs: which builds are instrumented, how data from unit, integration and long-running runs is merged, and why the artefact that ships is never the instrumented one.

## Coverage is measured, not deduced No tool can tell you from the source alone which lines a suite will execute; that is the halting problem wearing a hat. Coverage is therefore always an observation of a real run, and the mechanism behind every coverage report is the same: something was added to the code so that execution leaves a trace. ### Probes and basic blocks The unit of instrumentation is the **basic block** — a straight-line run of instructions with one entry and one exit, ending at a branch, a call that can divert control, or a return. Inside a basic block, either everything runs or nothing does, so one probe per block is enough. A probe is tiny: set a flag, or increment a counter, in a table indexed by probe id. That is why probes are placed at block boundaries rather than on every line. A line-per-probe scheme would be both slower and wrong, because several lines can share a block and a single line can span several. Branch data falls out of the same placement. If the block on the true side of a decision was hit and the block on the false side was not, that decision has one of two outcomes covered. No extra machinery is needed for branch coverage beyond knowing which blocks belong to which decision. ### Three places the rewriting can happen 1. **Source instrumentation.** The tool rewrites the source (or an intermediate form of it) before compilation, adding probe statements. Simple to reason about; requires a separate instrumented build, and the compiled output is not the artefact you ship. 2. **Offline artefact instrumentation.** The compiled units are rewritten as a build step, producing an instrumented copy that the tests run against. The shipped artefact stays clean because you ship the un-instrumented build. 3. **On-the-fly instrumentation.** An agent attaches at process start-up and rewrites each unit as it is loaded. Nothing on disk changes, which makes it convenient in a pipeline — and it introduces the classic failure mode below. A fourth approach, driving the runtime's debugging interface to trace execution, avoids rewriting entirely but is far slower and is not what production coverage tooling normally does. ### The lifecycle of the data During the run, probes update an in-memory table. On normal termination, or when something explicitly asks for a dump, the table is serialised to an execution-data file: probe ids and hit counts, and nothing human-readable. A later report step joins that file against the artefact and its retained **line-number information** to translate probe ids into files, lines, branches and functions. This is why coverage reports go wrong or vanish when a build strips debug information, and why the report step needs both the data file and the exact artefact that produced it. Because the data is just counters keyed by probe id, files from several runs can be **merged**: a unit-test run, an integration run and a long-running exploratory session can be summed into one report, as long as they instrumented the same artefact. ### What goes wrong, and the failure that catches people On a clinical-trial data-capture module, the integration suite is killed by an intermittent timeout after 9 minutes 40 seconds, part-way through the second-to-last file. The unit run merges normally, but the integration run never reached its dump, so its buffered probe hits are gone. The merged report shows branch coverage on the submission path falling from 74.6% to 41.2%, and on a 3-week release train that looks exactly like a regression somebody introduced. Nothing changed in the tests; the process simply died before it wrote its data. The tell is that the missing coverage is contiguous and follows the run order, not the diff. The rest of the standard traps: - **Separate processes.** If the tests drive a service running in another process, that process must itself be instrumented and must dump its data — usually on shutdown or through an explicit collection step. Nothing crosses the process boundary by magic. - **Forked children.** Work done in child processes is invisible unless each child dumps its own data file. - **Loaded too early.** With an on-the-fly agent, anything loaded before the agent is in place is never rewritten, so it reports as entirely uncovered no matter how hard the tests hammer it. - **Generated and synthetic code.** Compiler-generated members and generated sources land in the denominator and drag the percentage around for reasons that have nothing to do with the tests. - **Timing.** Probes cost something. The effect is usually modest, but timing-sensitive and concurrent tests can behave differently under instrumentation, which is one more reason not to ship an instrumented build. ### The one-line summary for an interview Probes at basic-block entries, inserted by rewriting the source or the compiled artefact; counters dumped at exit; a report step that maps probe ids back to lines using retained line information; and a family of failure modes that all reduce to "the data never got written, or the code was never instrumented in the first place".

  • Why can a report fail to attribute coverage to source lines even though the run produced execution data?
    Because the execution data holds only probe ids and counts. Turning those into files and line numbers needs the line-number information the compiler kept in the artefact, plus the very artefact the probes were cut for. A build that strips debug information, or a report run against a recompiled artefact whose probe numbering has shifted, gives either no line attribution or attribution that is quietly wrong.
  • Your integration tests drive a service in a separate process. How does that service's coverage reach the report?
    The service process must be started with instrumentation of its own and must write its own execution data — typically on graceful shutdown, or through an explicit collection step while it is still alive. That file is then merged with the test process's file before reporting. If the service is killed rather than shut down, the data is lost, so integration coverage pipelines usually make graceful shutdown part of the test teardown.
  • What does instrumentation do to the behaviour of the code under test?
    It adds work at every basic-block entry, so runs get slower and code size grows. For most suites the effect is unremarkable, but timing-sensitive and concurrent tests can pass or fail differently under instrumentation, and a race may appear or disappear. That is one reason the instrumented build stays inside the pipeline and the artefact that ships is the clean one.

saying these in an interview costs you the question

  • Thinks coverage is computed statically without running tests
  • Believes a sampling profiler's output is coverage data
  • Assumes a separate process's coverage appears automatically
  • Says instrumentation cannot affect timing or test outcomes
  • Expects complete data from a process killed mid-run
  • Thinks probe ids map to lines without line information

context

resolving links…