A nightly Newman run reports twenty passing iterations but the data file holds eight rows — how do you diagnose that?
answer
- Two inputs, nobody reconciles them
- Green does not mean covered
- A leftover flag outlived the file
- Look at the last record, repeatedly
basics
~20 sAn iteration count pinned above the data file's row count is the usual cause: iterations past the last row replay that row, so the run stays green while covering nothing new. Drop the count and let the rows govern.
solid answer
~50 sThe count and the file are independent inputs and nothing reconciles them. A pinned `-n` larger than the number of rows in `-d` still performs every requested iteration, and each one past the last row is handed **that last row again** — no wrap, no error, no marker in the output. So twenty green iterations over eight rows means twelve passes re-tested the eighth record. Confirm it by logging `pm.iterationData.toObject()` alongside `pm.info.iteration` and looking for the same record at consecutive positions, then check where the count is coming from — usually a pipeline template that was written before the file grew, or a debugging value that never got removed. The durable fix is to stop passing a count at all when passing a data file, so the row count governs and the two can never disagree.
code
javascript · 5 linesconst seen = pm.iterationData.get('customerId');
console.log('iteration', pm.info.iteration, 'of', pm.info.iterationCount, 'customer', seen);
if (pm.info.iteration > 0 && seen === pm.variables.get('previousCustomerId')) {
throw new Error('data exhausted: iteration is replaying the last row');
}go deeper
Know that the iteration count and the data file are separate inputs, so a run can report more iterations than the file has records without anything being broken.
Explain the mechanism behind the symptom: rows are consumed in order, then the last one repeats, with no wrap and no error to signal the change.
Show the diagnosis and the durable fix — log the record with its position, find where the pinned count came from, and remove the second source of truth rather than patching around it.
Own the policy: which run parameters a shared pipeline template may fix, and how suites are required to evidence coverage instead of inferring it from a passing verdict.
## The symptom and why it is deceptive A scheduled run finishes, every iteration passes, and the iteration total is comfortably larger than the number of records the team believes it has. Nothing about that output is malformed: the requests really executed, the assertions really passed, the timings are real. The problem is that a majority of those iterations exercised the **same** record, so the run's apparent breadth is fiction while its verdict is honest. This is the failure mode that survives review longest, because every signal you would normally check is green. ## What actually happened - The run was given both an explicit iteration count and a data file, and the two disagree. - Rows are consumed in order until the file is exhausted. - Every iteration after that is handed the **last row again** — the runner does not wrap back to the first row and does not raise an error. - The repeated iterations are indistinguishable from real ones at the row level: the scope answers normally, no column is missing, nothing is blank. - The positional members keep advancing exactly as they would for genuine rows, so `pm.info.iteration` climbing to the requested total proves nothing about coverage. The root cause is almost never mysterious once you look: a pipeline template that hardcoded a count when the file was shorter, a value left behind from a debugging session, or two people who each assumed the other flag was authoritative. ## Confirming it 1. **Log the record with the position.** Print `pm.info.iteration` next to `pm.iterationData.toObject()` and read the tail of the output: the same record appearing at several consecutive positions is the whole diagnosis. 2. **Compare the numbers directly.** `pm.info.iterationCount` reports the plan, so put it beside the number of rows the file actually holds; they should be equal and here they are not. 3. **Inspect the invocation, not the collection.** The collection document declares nothing about a data file, so the disagreement is entirely in the command line that launched the run. 4. **Reproduce locally** with the same file and the same count, then again with the count removed, and confirm the iteration total drops to the row count. ## The fix, and when a count is still right | situation | what to pass | why | |---|---|---| | every row must run once | the data file, no count | the row count governs; the flags cannot disagree | | repeat one request to expose flakiness | a count, no data file | repetition is the point and there is no per-row variation | | a deliberate short debugging pass | a smaller count, temporarily | acceptable only while you are watching it | | both flags, permanently pinned | avoid | coverage silently inflates or truncates as the file changes | The durable answer is the first row of that table. Dropping the count is not a workaround; it removes the second source of truth entirely, which is the only way the mismatch stops being possible. ## Making the run say it out loud If your situation genuinely requires both flags, make the run prove its own coverage rather than trusting the configuration: - Give every record a distinct identifying column and assert in a script that the value differs from the one the previous iteration saw. A repeat then fails loudly at the exact pass where the data ran out. - Carry the expected number of records in the data itself — a column repeated on every row — and compare it against `pm.info.iterationCount` on the first pass, failing immediately when they disagree. - Log the record on every iteration unconditionally. It costs nothing and turns any future recurrence into a two-second read of the output. ## What not to conclude - **Do not conclude the file was misread.** The rows were consumed correctly; the run was simply asked for more passes than there were records. - **Do not conclude the runner wrapped.** It repeated the final row, which looks similar in a summary line and is completely different in the output. - **Do not add retries or padding rows** to make the numbers line up. Padding the file with duplicate records reproduces the same fiction by hand, and retries change what the run means. - **Do not treat a green result as coverage evidence.** In this failure mode green is precisely what you get; coverage has to be asserted, not inferred from the absence of failures. The interview value of the scenario is that it is not a bug in anything. Two independent inputs were both honoured exactly as specified, and the damage came from nobody owning the relationship between them — which is the shape of most production configuration incidents.
- The pipeline template you inherited hardcodes the iteration count for every suite. What do you change?Remove the count from the template and let each suite's data file govern, keeping the count as an opt-in for suites that repeat a run without data. A template-level constant guarantees that every file which later grows or shrinks silently drifts from it, and the drift shows up as either lost coverage or repeated records — never as a failure.
- How would you prove to a reviewer that the run really covered every record?Have the run assert it rather than claim it: log a distinct identifier per iteration and fail when consecutive iterations repeat one. The output then enumerates the records actually exercised, and a repeat becomes a failure instead of a line nobody reads. Counting passing iterations proves only that the requested number of passes ran.
saying these in an interview costs you the question
- Blames the data file parser for the extra iterations
- Says the runner wrapped around to the first row
- Adds duplicate rows to make the counts line up
- Treats an all-green run as proof of full coverage
- Looks in the collection document for the data file setting