skip to content

Which column in a JMeter JTL file must a pipeline's pass rule count?

level: middleimportance: must knowfreq 58%

answer

  1. One column carries the verdict per row
  2. Response codes mislead once assertions run
  3. Ask what fills the message column at all
  4. Empty message does not mean passing sample

basics

~20 s

The success column. JMeter writes the literal true or false there for every sample, and a false row is the only dependable marker of a failed sample; responseCode can still read 200 on a failed one.

solid answer

~40 s

Count `false` in the **`success`** column. JMeter writes `SampleResult.isSuccessful()` there as the literal `true` or `false`, and `jmeter.save.saveservice.successful` defaults to `true`, so the column is present unless somebody switched it off. Do not gate on `responseCode`: an assertion that fails a body check flips `success` to `false` while `responseCode` stays `200`. Do not gate on `failureMessage` either — `CSVSaveService` fills it from `SampleResult.getFirstAssertionFailureMessage()`, so it is **empty** for a connection failure or a plain HTTP 500 with no assertion attached, and carries only the *first* message when several assertions failed on one sample. A run where every sample was refused writes 1,200 `false` rows and 1,200 empty failure messages.

code

text · 3 lines
text
timeStamp,elapsed,label,responseCode,responseMessage,threadName,dataType,success,failureMessage,bytes,sentBytes,grpThreads,allThreads,URL,Latency,IdleTime,Connect
1757404801234,15,POST /checkout,Non HTTP response code: java.net.ConnectException,Non HTTP response message: Connection refused,tg 1-4,text,false,,1284,0,20,20,https://shop.example/checkout,0,0,0
1757404801250,142,POST /checkout,200,OK,tg 1-7,text,false,Test failed: text expected to contain /orderId/,318,412,20,20,https://shop.example/checkout,110,0,12

go deeper

for a junior

Know that the results file has a success column holding true or false, and that this is the field to look at when asking whether a sample passed.

for a middle

Explain why responseCode and failureMessage both mislead: an assertion failure keeps a 200 code, and the message field only carries the first assertion message, if any.

for a senior

Show that you resolve the column by header name rather than position, filter sub-samples out of the ratio, and treat an empty message as no information rather than as a pass.

for a principal

Set the convention: one shared rule reading the success column by name across every team's plans, so error rate means the same thing in every pipeline that reports it.

## The three columns people confuse The default CSV header a `-l` file starts with is: ``` timeStamp,elapsed,label,responseCode,responseMessage,threadName,dataType,success,failureMessage,bytes,sentBytes,grpThreads,allThreads,URL,Latency,IdleTime,Connect ``` Three of those look like they carry the verdict, and only one of them does. | Column | Filled from | Why a gate should or should not read it | |---|---|---| | `success` | `SampleResult.isSuccessful()`, written as `true`/`false` | **This is the verdict per sample.** Failed sampler *or* failed assertion, both land here | | `responseCode` | the protocol's own code | A wrong-but-200 body assert-fails with `responseCode` still `200`; a non-HTTP sampler may put anything here | | `failureMessage` | `getFirstAssertionFailureMessage()` | Empty unless an **assertion** produced a message, and then only the first one | ## Why failureMessage is the trap `CSVSaveService.resultToDelimitedString` writes the failure column with `sample.getFirstAssertionFailureMessage()`, appending an empty field when that returns `null`. `SampleResult.getFirstAssertionFailureMessage()` walks the sample's `AssertionResult[]` and returns the first non-null message it finds. Two consequences fall straight out of that: - **A sample can fail with no message at all.** A connection refused, a socket timeout, a bare HTTP 500 with no assertion attached to the sampler — `success` is `false`, `failureMessage` is empty. This is exactly the run where every sample failed: a pipeline rule written as *fail if any row has a non-empty failureMessage* sees nothing and goes green, which is the false pass this leaf exists to describe. - **Several failing assertions collapse to one message.** Only the first non-null message survives into the row, so the column is a diagnostic hint, never a count of what failed. The column is also optional: `jmeter.save.saveservice.assertion_results_failure_message` defaults to `true`, but a plan or a properties file that turns it off simply drops the field from the header, and a rule addressing columns by position then reads the wrong one. ## Writing the rule so it survives 1. **Resolve the column by header name, not by position.** The JTL layout is property-driven: `jmeter.save.saveservice.*` keys add and remove fields, so `success` is only the eighth field in a default configuration. Read the first line, find the index of `success`, then count. 2. **Count rows, not bytes.** The error rate you want is `false` rows over total sample rows. The header line is not a sample; a run that sampled nothing still leaves a one-line file, because `jmeter.save.saveservice.print_field_names` defaults to `true`. 3. **Decide what a sub-sample means.** Embedded resources and transaction children can appear as rows too, so a naive ratio mixes page samples with their children. Filter by `label` if the rule is meant to be about named transactions. 4. **Do not re-derive success from `responseCode`.** That throws away every assertion the plan authors wrote, which is the whole reason the assertions are there. ## The one-line version The `success` column is JMeter's own answer to "did this sample pass", already combining the sampler's outcome with every assertion in scope. A pipeline gate that counts anything else is guessing at a verdict JMeter has already written down.

  • A sample has three failing assertions. How many messages reach the JTL row?
    One. CSVSaveService writes getFirstAssertionFailureMessage(), which returns the first non-null failure message in the sample's AssertionResult array and stops. The row still reads success=false, so the count of failures is right even though the diagnosis is partial.
  • What happens to a rule that addresses the success column by field number?
    It breaks the moment a jmeter.save.saveservice.* key is flipped, because those properties add and remove JTL fields. success is only the eighth field in the default layout; a rule should read the header line and resolve the index by name.

saying these in an interview costs you the question

  • Counts HTTP 500s and calls that the error rate
  • Gates on failureMessage being non-empty
  • Thinks responseCode 200 always means a passing sample
  • Assumes every failing sample records a failure message
  • Believes the JTL writes one row per assertion