skip to content

A Gatling run that declared no assertions and one whose assertions all held both exit 0. How would you stop the first from being read as a pass?

level: principalimportance: should knowfreq 30%

answer

  1. Nothing judged looks like everything passed
  2. The simulation must prove its own work
  3. A request-count floor an empty run fails
  4. Console prints nothing when nothing asserted
  5. Enterprise JUnit step fails without assertions

basics

~20 s

A run with no assertions and a judged passing run both exit 0, so the simulation must carry the proof — typically an assertion an empty or cut-short run cannot satisfy, such as a request-count floor.

solid answer

~50 s

Gatling's exit code answers "did anything I asked about turn out badly?" and never "did I ask about anything?", so the distinction has to be created somewhere. Three places are available, and they are not equivalent. You can **observe** it — Gatling prints one line per assertion, so a green run that printed none judged nothing — but an observation is not a gate. You can **enforce it outside the run**, which works but lives in whichever job runner you happen to use and does not travel. Or you can put it **inside the simulation**: register an assertion that an empty or truncated run cannot satisfy, such as a floor on the total request count, so the artefact under test proves its own work happened. Deciding *what* a latency or error-rate target should be is a separate question; this one is only about proving a judgement occurred at all.

code

java · 22 lines
java
import io.gatling.javaapi.core.*;
import io.gatling.javaapi.http.*;

import static io.gatling.javaapi.core.CoreDsl.*;
import static io.gatling.javaapi.http.HttpDsl.*;

public class GatedSimulation extends Simulation {

  private final HttpProtocolBuilder httpProtocol = http.baseUrl("https://example.com");

  private final ScenarioBuilder scn =
      scenario("browse").exec(http("home").get("/"));

  {
    setUp(scn.injectOpen(constantUsersPerSec(10).during(60)))
        .protocols(httpProtocol)
        .assertions(
            // proves the run actually did the work, not just that it ended
            global().allRequests().count().gte(500L),
            global().failedRequests().percent().lte(1.0));
  }
}

go deeper

for a junior

Understand that a passing exit code does not prove the run judged anything, and that the simulation itself is where assertions are declared. Ask what a green run actually checked.

for a middle

Be able to propose a concrete mechanism, such as asserting a floor on the request count, and explain why an empty or truncated run cannot satisfy it.

for a senior

Weigh a gate inside the simulation against one in the job that launches it, and say which failures each catches and which it misses when the runner changes.

for a principal

Own the position: decide where the safeguard lives across a whole suite, name the trade-off you accepted, and say what evidence would make you move it.

This is a design question rather than a lookup, and it is the one worth having an opinion on. Gatling's exit code is a verdict over the assertions a simulation registered. If it registered none, the fold over an empty list is vacuously true and the process exits `0` — the same integer a fully-judged passing run produces. Nothing in Gatling closes that gap for you, so the question is where you choose to close it. ## The three places it can be closed ### 1. Inside the simulation Register an assertion that a run which did nothing cannot satisfy. A floor on the number of requests is the canonical one: assert that the total request count is at least some number the run must reach if it actually generated its load. - **It travels.** It is part of the simulation source, so it holds however the simulation is launched — a developer's machine, a build job, a different runner entirely. - **It catches more than the empty case.** A run cut short *gracefully* — `maxDuration` reached, or `stopLoadGenerator` called from the scenario — still finishes, still has its assertions evaluated, and so its shortfall against the floor surfaces as `2`. Two cases it does **not** catch: a run pointed at a host it could not reach, because connection-refused, unknown-host and timeout attempts are all still recorded as requests and the total-request metric counts every request whatever its status, so that run clears the floor exactly as a healthy one does — the error-rate assertion on the next line of the example is what catches it; and a *crashed* run, which fails the run's outcome before any assertion is evaluated, exiting `1` with no verdict at all. - **It costs a judgement.** The floor has to be low enough not to be flaky and high enough to be meaningful, which is a decision someone has to make and revisit when the profile changes. ### 2. Around the run Fail the job when the run produced no verdict. Gatling prints one line per assertion with its description, its outcome and the actual measured value, so the absence of those lines is a reliable observable. - **It needs no change to the simulation** and covers every simulation at once. - **It does not travel.** It lives in whatever launches the run, and a simulation run any other way is unprotected. - **It is brittle against output.** Anything keyed to console text is coupled to formatting you do not own. ### 3. In a second artefact On Gatling Enterprise, the CI plugins leave a JUnit-format XML result directory, `gatlingEnterpriseJunitResults`, in the job workspace, with the run's assertion results in it, and a JUnit publishing step reads it. What makes this interesting is the documented behaviour when there is nothing to publish: Gatling's own integration pages warn, in as many words, that **if the simulation declares no assertions the JUnit step will fail.** That is the inverted signal, and it is worth stating plainly: | surface | run with no assertions | |---|---| | process exit code | `0` — reads as a pass | | JUnit results publication | **fails the step** | The same condition produces opposite signals depending on which surface the job reads. Community Edition has no equivalent artefact, so this route is only available where Enterprise is. ## How to choose 1. **Start inside the simulation.** A gate that lives in the artefact under test is the only one that cannot be left behind by a change of runner, and it is the version that still protects a developer running the simulation by hand. 2. **Add the outer check if you own many simulations.** One rule covering a whole suite is cheaper than one assertion per simulation, and it catches the simulation written next week that nobody remembers to gate. 3. **Use the second artefact where you have it**, but do not rely on it as the only mechanism — it is available on one edition and configured per runner. They compose. The inner assertion is the floor; the outer check is the sweep; the artefact is the audit trail. ## What this is not Two adjacent questions are deliberately out of scope here. **Which** target a pass rule should assert — a latency percentile, an error rate, a throughput number — and what value it should take is a separate discipline, and so is deriving a regression threshold from measured run-to-run noise. This question is narrower and comes first: before you argue about the right threshold, make it impossible for a run to sail past with **no threshold at all**. ## The argument to be ready for Someone will say the empty-assertion case is a code-review problem, not an engineering one — just review for it. That is a reasonable position and it is also how the problem got here: the failure is silent, compiles, and passes. The counter-argument is that reviews catch what reviewers look for, and nothing about a missing `assertions(...)` call draws the eye. A mechanism that fails loudly does not depend on anyone remembering. Whichever side you land on, say which, and say what evidence would change your mind.

  • Why is a request-count floor preferable to checking the console output for assertion lines?
    Because it is part of the simulation rather than part of whatever launched it. It holds on a developer's machine and under any runner, it is reviewed alongside the scenario, and it also catches a run that was cut short gracefully — `maxDuration` reached, or `stopLoadGenerator` called — because such a run still reaches assertion evaluation with a request count below the floor. A console check protects only the one place it is wired into. Neither mechanism catches a crashed run: that one never gets as far as the assertions and exits `1`.
  • On Gatling Enterprise, what happens to a JUnit publishing step when the simulation declares no assertions?
    It fails. The Enterprise CI plugins write a JUnit-format XML results directory carrying the assertion results, and Gatling's own integration documentation states that with no assertions in the simulation the JUnit step will fail — the exact opposite of the `0` the exit code reports for the same run.
  • Does a request-count floor risk becoming a flaky gate?
    It can, if it is set close to the expected volume. Pick a value well below what a healthy run produces but far above what a broken one does, and revisit it whenever the injection profile changes. Its job is to detect "this run did nothing", not to police throughput.

It is the difference between a smoke alarm that reports no smoke and one whose battery is flat. Both are silent, and only one of them is telling you something.

saying these in an interview costs you the question

  • Treating a green Gatling run as evidence a judgement happened
  • Relying only on code review to catch a missing assertions call
  • Assuming Gatling fails a run that asserted nothing
  • Building the only safeguard into one job runner
  • Setting a request-count floor so high it becomes flaky