skip to content

How do you report exploratory testing progress when there is no test-case count?

level: seniorimportance: should knowfreq 49%

answer

  1. Count the unit you actually have
  2. The empty rows are the message
  3. Effort applied, not risk retired
  4. Say why the missing sessions went missing
  5. Every number travels with its caveat

basics

~20 s

Report in sessions and areas: sessions completed against sessions planned, which product areas have had sessions and which have not, roughly how much of the time stayed on charter, and the obstacles currently slowing sessions down.

solid answer

~50 s

The session is the unit, so it is what you count. A readable report has four parts. **Sessions**: completed versus planned, this week and cumulatively. **Areas**: a simple grid of product areas against sessions run, which makes the untouched rows visible — that is the honest coverage statement when no case list exists. **On-charter share**: how much of the time went to the agreed missions versus incidental discoveries, since a collapsing on-charter share usually means the missions were wrong. **Obstacles**: the issues currently making sessions expensive, with who can clear them. Deliberately absent are counts of defects presented as an achievement, and any implication that finishing the planned sessions means an area is safe. Sessions measure effort applied and where it was applied; they never measure how much of the product's risk has been retired, and saying so plainly is what stops the number from being misread.

go deeper

for a junior

Know that progress is reported in sessions and areas rather than test cases, and that finishing the planned sessions is a statement about effort applied, not a statement that the area is now safe.

for a middle

Be able to build the report: sessions completed against planned, an area grid whose empty rows are the point, the on-charter share, and the obstacles list. Explain why defect counts make a poor progress measure.

for a senior

Show judgement under pressure from a stakeholder who wants one number. Give the figure, attach its limit in a sentence, and be able to explain what a falling on-charter share or a week of lost sessions actually tells you.

for a principal

Own how this reporting sits beside the team's other numbers without being quietly converted into a completeness claim. Be ready to say what you would report to an executive audience and what you would refuse to.

### The problem A scripted approach reports itself: 412 cases, 380 run, 366 passed. It is a badly flawed measure of anything that matters, but it is legible, and legibility is why leads reach for it. Exploratory work has no such number, and the first instinct — reporting hours, or defect counts — produces something worse. Hours say nothing about what was looked at. Defect counts reward the areas that happen to be broken and punish a tester who spends a week confirming that a risky area is solid. The technique's answer is to report the unit it created: the session. ### The four things worth reporting **Sessions completed against sessions planned.** "Nine of fourteen planned sessions run against checkout this week." This is a statement about effort delivered, and it is honest as long as nobody dresses it up as a statement about quality. It has the useful property of exposing its own shortfall: if only nine of fourteen ran, the missing five have a reason, and the reason is usually in the obstacles. **An area-by-session grid.** List the product areas the team reports in — down one side, sessions across. What the grid shows best is emptiness. In a rebuilt bookstore checkout with 23 named areas, if 6 of them have had no session at all, the grid says so at a glance, and no defect count or percentage can convey that. This is the closest thing exploratory work has to a coverage claim, and its honesty comes from being explicitly about *attention applied*, not about anything measured inside the product. **The on-charter share.** Aggregated across sessions, this says how much of the week's testing went to the missions the team agreed. A high share means the plan matched reality. A share that has fallen to 55% is not misbehaviour; it usually means the charters were written against the wrong model of the product and the testers are following the real risk. Either way it is a prompt for a conversation, not a rebuke. **The live obstacles.** The issues that are currently making sessions expensive, and who owns each. This is the part of the report a lead can act on this week. ### A worked example An online bookstore is releasing a rebuilt checkout. The report reads: eleven sessions planned this week, eight run; three lost because the payment sandbox was unavailable for a day and a half. Areas: gift-card redemption, address validation and split shipments have each had sessions; currency and locale handling, and the confirmation email, have had none. On-charter share about 70%, dragged down by a clock-difference finding in gift-card expiry that pulled two sessions sideways into time handling. Obstacles: no way to seed gift cards with past expiry dates, and no authoritative answer on which service owns time. One incidental observation is parked for a session of its own — the confirmation page is missing its stated budget of 2.4 seconds at the 92nd percentile. Every line of that is checkable and none of it claims more than it knows. A reader learns what was looked at, what was not, why three sessions vanished, and what to fix so next week goes better. ### What not to report, and what to say when pressed Do not report defect counts as an achievement, because it makes the tester's output a function of how broken the code happens to be. Do not report a percentage that implies completeness — "checkout testing is 68% done" — because the denominator does not exist and inventing one converts a good report into a false one. Do not let "all planned sessions completed" be read as "this area is safe"; sessions are effort, and the judgement about whether an area has had enough belongs to a person looking at the risk, not to a counter reaching a target. When a stakeholder insists on one number anyway, the productive move is to give them the sessions-completed figure with the areas grid attached and to state its limit out loud, in one sentence, every time it is reported: this says where we spent attention, not how much risk is left. A number that travels with its own caveat survives; a number that does not gets quoted back to you in a release meeting as a guarantee.

  • A stakeholder asks what percentage of the exploratory testing is finished. How do you answer?
    By refusing the denominator politely and offering a better answer. There is no fixed population of exploratory work to be a percentage of, so any figure would be invented. Instead give sessions run against sessions planned for the current risk picture, name the areas that have had no session at all, and say what would have to happen for you to want more sessions. That answers the real question — where are we exposed — rather than the one that was asked.
  • Two testers ran sessions on the same area this week. Does that count as one area covered or two?
    Two sessions, one area, and the grid should show the count rather than a tick. Repeat sessions on one area are often exactly right — a second tester with a different model finds things the first could not — but the report must not let two sessions on a familiar area look like coverage of two areas. The grid measures attention applied, so it should carry counts, not booleans.
  • How do you report a week in which sessions found almost nothing?
    As a normal, informative week. Sessions applied attention to named areas and surfaced little, which is evidence about those areas and is worth exactly as much as a week of findings. The failure mode to avoid is a reporting culture in which a quiet week looks like a wasted one, because that pressure pushes testers toward areas likely to yield bugs rather than areas that carry risk.

saying these in an interview costs you the question

  • Reporting defect counts as a productivity measure
  • Quoting a percentage complete with an invented denominator
  • Treating all planned sessions run as proof an area is safe
  • Reporting hours spent instead of sessions and areas
  • Hiding sessions that were lost to blocked environments
  • Marking an area covered after a single session

context