skip to content

Can a JMeter build step decide pass or fail from the summariser's Err count?

level: middleimportance: nice to knowfreq 26%

answer

  1. Ask what type the Err figure really is
  2. Two line shapes, one interval and one total
  3. Numbers are padded to fixed widths
  4. A run with nothing sampled prints zero errors

basics

~20 s

Only fragilely. JMeter's summariser prints Err as padded text on stdout and into jmeter.log; it is a report line, not a status, and a run that sampled nothing prints Err with zero exactly like a clean run does.

solid answer

~40 s

The summariser is on by default because `summariser.name=summary` is one of the few keys the shipped `bin/jmeter.properties` actually sets. Every `summariser.interval` seconds — 30 by default — it prints a `summary +` delta line, and at the end one cumulative `summary =` line, to both stdout and the log (`summariser.out` and `summariser.log`, both `true`). `Err:` is the count of samples whose `isSuccessful()` was false, followed by that count over total samples formatted as `#0.00%`. A build *can* grep it, but the numeric fields are right-aligned into fixed widths, the delta and total lines look nearly identical, and a run that produced no samples still prints a total line reading zero errors. Parse the JTL instead.

code

text · 2 lines
text
summary +    600 in 00:00:30 =   20.0/s Avg:   112 Min:    45 Max:   980 Err:   600 (100.00%) Active: 20 Started: 20 Finished: 0
summary =   1200 in 00:01:00 =   20.0/s Avg:   108 Min:    41 Max:   980 Err:  1200 (100.00%)

go deeper

for a junior

Know that the summary lines JMeter prints during a non-GUI run include an Err count, and that these lines are printed text rather than anything the build system reads automatically.

for a middle

Describe the two line shapes and the padding, and explain why a zero error count says nothing until you have also checked how many samples were taken.

for a senior

Argue for gating on the JTL and keeping the summariser as narration, and name the cases where the two counts legitimately disagree so a mismatch is not chased as a bug.

for a principal

Decide what a load stage's job log must contain for an on-call engineer to diagnose it without re-running, and keep that requirement separate from what the gate parses.

## What the line actually contains `Summariser.format()` builds one line per report. Reading the builder straight off the source, a cumulative line is the summariser's name, then `=`, then the sample count right-aligned to six characters, `in` plus the elapsed time as `%02d:%02d:%02d`, `=` plus the rate right-aligned to six with one decimal, then `/s Avg:`, `Min:` and `Max:` each right-aligned to five, then `Err:` right-aligned to five, then the error share in brackets formatted as `#0.00%`. Interval lines carry `+` instead of `=` and append `Active:`, `Started:` and `Finished:` thread counts. For the run this leaf is about — twelve hundred samples, every one refused — that is: - an interval line: `summary + 600 in 00:00:30 = 20.0/s ... Err: 600 (100.00%) Active: 20 ...` - a final line: `summary = 1200 in 00:01:00 = 20.0/s ... Err: 1200 (100.00%)` The information is right there in the job log. Nothing reads it. ## Why grepping it is fragile 1. **It is padded text, not a field.** `longToSb` right-aligns into fixed widths, so `Err: 0`, `Err: 600` and `Err: 1200` differ in whitespace. A pattern like `Err: 0` matches nothing on a wide count and matches accidentally on some others; the pattern has to tolerate runs of spaces. 2. **Delta and total lines both exist.** `+` lines are per-interval and `=` is cumulative. Grep for `summary` and you capture both, and on a long run there are many of the former. 3. **Zero samples reads as zero errors.** `getErrorPercentage()` returns `0.0` when the counter is zero, so a run that never sampled anything prints a total line with no errors — indistinguishable from a perfect run to a rule that only looks at `Err:`. 4. **It goes to two places.** `summariser.out` writes to stdout and `summariser.log` writes to the log, both `true` by default; a CI step that captures only one, or captures both, changes what the grep sees. 5. **It can be switched off entirely.** Comment out `summariser.name` and the lines disappear, and a gate that greps for an absent line reports success. That failure shape — a check whose *input* went missing and which therefore passed — is the same one this leaf's whole topic turns on. ## Where the line is still worth reading - **As a human signal in the job log.** `Err: 1200 (100.00%)` on the last line is the fastest way to see, by eye, that a green build was a lie. - **As a live progress indicator.** The interval lines and their `Active`/`Started`/`Finished` counts tell you the run is moving while it is still running, which the JTL cannot do until it is flushed. - **As a cross-check.** If your JTL-based gate and the summariser disagree about the error count, something is filtering rows — usually sub-samples or a transaction controller, and `summariser.ignore_transaction_controller_sample_result` defaults to `true`, which is one place the two counts legitimately differ. ## The verdict Treat the summariser as the run's narration and the JTL as its record. A pipeline should gate on the record: the JTL has one row per sample with a `success` field you can count without guessing at whitespace, and it survives the job log being truncated or rotated. Keep the summariser on, print it, read it when a build goes red — just do not make it the thing that decides.

  • How would a build tell a JMeter run that sampled nothing from one that sampled cleanly, using only the summary line?
    By reading the sample count, not the Err count. Both runs print zero errors, but the clean run's count is the number you expected and the empty run's is zero or the line is absent entirely. Any honest gate needs a floor on samples as well as a ceiling on errors.
  • Why might the summariser's error count differ from a count of false rows in the JTL?
    Because the two see different sample sets. summariser.ignore_transaction_controller_sample_result defaults to true, so transaction controller results are left out of the summary, while the JTL may carry those rows and sub-samples as well. Filter the JTL the same way before comparing.

saying these in an interview costs you the question

  • Greps for the literal text Err: 0 and ignores the padding
  • Treats an absent summary line as a passing run
  • Cannot distinguish the interval line from the cumulative one
  • Assumes zero errors implies samples were actually taken
  • Thinks the summariser writes a file the pipeline can parse