Threshold Enforcement

## What the status code is actually about `jmeter -n -t plan.jmx -l results.jtl` is an ordinary JVM process, and its exit status answers one question: **did the process finish its work without an unhandled error?** It never answers *did the system under test behave?* A run in which all 1,200 samples came back `Connection refused` ends exactly the way a clean run ends — the engine started, the threads ran their loops, the listeners were told the test had ended, JMeter printed `... end of run`, the JVM returned `0`, and the build step after it went green. Three weeks of that is a real and common failure: the nightly job is green, the dashboard is being generated, and nobody has looked at the `Err:` figure. ## The only two non-zero paths in JMeter 6.0.0 Grep the shipped sources and the CLI has exactly two places that hand back a failing status, and neither of them looks at a `SampleResult`: 1. **A throwable escaping start-up.** `JMeter.start(String[])` wraps the whole CLI path in a `catch (Throwable)` that logs `An error occurred: ` and calls `System.exit(1)`. A `-t` path that does not exist, a plan that will not deserialise, an output folder JMeter refuses to write — all real non-zero exits, and all of them about *arguments*, not results. 2. **A shutdown that got stuck.** When a test has been explicitly asked to stop *now* and `StandardJMeterEngine` still finds live threads after its wait, it logs `One or more test threads won't exit; see log file.`, prints `Fatal error, could not stop test, exiting` and calls `System.exit(1)`. The `jmeterengine.stopfail.system.exit` property gates that one call, and it is a report about a wedged JVM. Everything else exits `0`. That includes the case candidates find hardest to believe: a **test-tree compile error**. `StandardJMeterEngine.run()` traverses the plan with `PreCompiler` inside a `try`; on a `RuntimeException` it logs `Error occurred compiling the tree:` and simply returns. No listener is ever told the test started, so the `-l` results file is never even opened — and the process still exits cleanly. A pipeline that reads only the status code cannot tell *ran perfectly* from *never ran at all*. ## There is no threshold to set The second half of the answer matters as much as the first. JMeter has no threshold language. There is no plan element that says "fail above 1% errors", no property in `bin/jmeter.properties` that turns an error rate into an exit status, and no CLI flag that does it. Assertions exist, and they do real work — they turn a wrong-but-200 response into a sample marked failed — but an assertion result lives in the sample, not in the process. It changes a row in the results file and, at most, what the engine does next inside the run. It never reaches the shell. ## Where a real verdict can come from | Signal | Where it lives | Can it fail a build on its own? | |---|---|---| | Process exit code | the shell | No — `0` unless start-up or shutdown broke | | `success` column | the `-l` JTL file | Yes, once a step counts it and exits non-zero itself | | Summariser `Err:` figure | stdout and `jmeter.log` | Only if a step parses that text | | Assertion results | rows inside the JTL | They mark rows; they do not stop the process | So the pipeline shape that actually gates is two steps, not one: run JMeter, then run a check that reads the artefact JMeter left behind and exits non-zero on its own judgement. Everything this leaf is about follows from that: the check has to name a column, it has to survive a run that produced no rows at all, and it has to be as maintained as the plan it guards. ## What to say in the room Answer the literal question first — the status reports the process, not the samples — then name the two exceptions, then say the consequence out loud: *a green JMeter step is not evidence that the system responded.* That last sentence is the one interviewers are listening for.