skip to content

Would you compute a JMeter run's pass or fail inside the plan or after it?

level: principalimportance: should knowfreq 42%

answer

  1. Ask which participant the shell can hear
  2. One placement can stop, the other can fail
  3. Consider the run that was cut short
  4. Assertions make rows honest, not builds red

basics

~20 s

Put the verdict after the run, on the JTL, and keep in-plan elements for marking samples and for cutting a doomed run short. Nothing inside a JMeter plan can set the exit status the pipeline actually reads.

solid answer

~50 s

Both placements are real and they do different jobs. **Inside the plan**, assertions mark samples as failed and a JSR223 Listener sees every result through its `prev` binding, so it can accumulate a count into `props`; the third-party `jp@gc - AutoStop Listener` can stop a run once an error threshold is crossed. What none of them can do is set the process exit code. **After the run**, a step that reads the `success` column of the JTL owns the exit code, re-runs against an archived file, and is one rule for every plan — at the cost of only seeing what the save configuration kept, and only after the load has been spent. The defensible split: assertions in the plan so the rows are honest, an optional in-plan stop as cost control, and the verdict outside.

go deeper

for a junior

Know that assertions belong in the plan and that they mark samples rather than fail builds, so something outside JMeter has to turn those marks into a pass or a fail.

for a middle

Explain what a JSR223 Listener can and cannot reach: every result through its prev binding and JVM-wide props, but never the process exit code the pipeline reads.

for a senior

Argue the split concretely, including why an in-plan stop truncates the evidence and why a tearDown-based verdict is skipped on exactly the runs that were cut short.

for a principal

Own the arrangement across teams: where the rule is versioned, how a plan change that renames transactions is stopped from silently changing what the rule counts, and what a green load stage is allowed to mean.

## What each placement can actually do **In the plan.** Assertions are the reason a `success` value means anything: without them a wrong-but-200 response is recorded as a pass. A `JSR223 Listener` is handed each result through its `prev` (also `sampleResult`) binding and can count failures into `props`, which is JVM-wide, so a later element can read the tally. The third-party `jp@gc - AutoStop Listener` — a `jpgc` add-on, not part of the Apache download — watches error rate, response time and other series and stops the run when a configured threshold is crossed. **After the run.** A step that reads the `-l` JTL, counts `false` in the `success` column and calls `exit 1` itself. That is the only one of the two that the shell can see. ## The three things that decide it 1. **Only the outside step owns the exit code.** The AutoStop Listener stops the run by setting a system property named `auto_stopped` and sending `StopTestNow` over UDP to the port from `jmeterengine.nongui.port`. That is a *stop*, not a *fail* — the process still ends `0`. The only in-plan way to force a non-zero status is to call `System.exit` from a script element, and that cuts the run off mid-flight: the engine never reaches `notifyTestListenersOfEnd`, so there is no final summariser line and no `-e` dashboard, and every sample the plan still had to take never happens. The JTL itself survives — `System.exit` runs JVM shutdown hooks and `ResultCollector.testStarted` registers one that closes every open results file — so what you trade for the verdict is the rest of the run and the report built from it, not the rows already written. 2. **An in-plan verdict can be skipped.** A tearDown Thread Group is a natural place to compute a tally from `props`, but it does not run when the test is stopped, and by default it does not run on a graceful shutdown either unless the Test Plan's *Run tearDown Thread Groups after shutdown of main threads* option is ticked. So the very runs that most need a verdict — the ones something cut short — are the runs where the in-plan verdict silently does not happen. 3. **An outside rule is re-runnable and shareable.** It reads an archived artefact, so you can re-evaluate last month's run under this month's rule, and one implementation guards every plan instead of each plan carrying its own copy of the logic in a script element that nobody reviews. ## What the outside rule gives up | | In the plan | After the run | |---|---|---| | Can set the process exit code | No | Yes | | Runs even when the test is cut short | Not reliably | Yes, on whatever was written | | Can stop the run early | Yes | No — the load is already spent | | Sees the live result object | Yes, including the body | Only the saved columns | | One rule across many plans | No, copied per plan | Yes | The honest cost of the outside rule is the last two rows of the left column: it cannot see anything the save configuration discarded, and it cannot stop a run that is already failing from burning an hour of a shared environment. ## The recommendation, and how to defend it Say the split out loud: **assertions in the plan, stopping in the plan, verdict outside the plan.** Assertions make the rows truthful. An in-plan stop is a cost control — it saves the environment and the wall-clock, and you accept that it produces a shorter results file. The verdict is a step of its own that reads the artefact and exits non-zero, because that step is the only participant in the whole arrangement that the pipeline can hear. Then name the second-order cost you are accepting: the rule now lives outside the `.jmx`, so it has to be versioned and reviewed like the plan, and a plan change that renames a transaction can quietly change what the rule counts. That is a real maintenance burden, and it is smaller than the burden of a verdict that cannot be expressed.

  • Why not just call System.exit(1) from a JSR223 element once the error count crosses a limit?
    Because it cuts the run off at an arbitrary point. The rows already written are not lost — System.exit runs JVM shutdown hooks and ResultCollector adds one that closes and flushes its files — but the engine never notifies its listeners of the end, so there is no final summariser line and no HTML dashboard, and every remaining sample is missing. You get a red build, a partial run, and a rule living in a script element nobody reviews.
  • What does the third-party AutoStop Listener change about a run's outcome?
    It changes how long the run lasts, not whether it passes. It sets a system property named auto_stopped and sends StopTestNow over UDP to JMeter's non-GUI port; the engine stops and the process still exits zero. Treat it as cost control and read the truncated JTL for the verdict.
  • What is the maintenance cost of keeping the rule outside the plan?
    The rule and the plan drift. A renamed transaction, a changed save configuration or a new sub-sample shape can change what the rule counts without anyone editing the rule. Version the two together and make a plan change that touches labels or saved columns require a look at the gate.

saying these in an interview costs you the question

  • Claims an assertion can make the JMeter process exit non-zero
  • Proposes System.exit from a script element as the normal gate
  • Believes a stopped run and a failed run are the same outcome
  • Assumes a tearDown group always runs to write the verdict
  • Puts the whole rule in a script element nobody reviews