skip to content

What are the four job outcome tests a ZAP automation plan can attach to a job, and how do they differ?

level: middleimportance: nice to knowfreq 38%

answer

  1. four kinds, one mandatory field
  2. three judge after, one judges during
  3. onFail picks the bucket
  4. monitor can end the job

basics

~20 s

A ZAP plan job can carry alert, stats, url and monitor tests. The first three are judged after the job finishes; monitor is judged while it runs and can stop a long job early. Each needs onFail.

solid answer

~40 s

Under a job's `tests` key a plan can assert on its outcome. **`stats`** compares a named statistic against a value with an operator; **`url`** asserts that a URL, and optionally regexes over its request or response, is present or absent; **`alert`** asserts the presence or absence of a finding from a given scan rule; **`monitor`** watches a statistic against a threshold **while the job runs** and stops the job when it is exceeded. Every test needs `onFail`, one of `warn`, `error` or `info` — it is mandatory, and a test without a valid one is dropped with a warning and never runs. `onFail` matters because it decides which bucket a failure lands in, and those buckets drive both the plan's early exit and its exit value.

code

yaml · 11 lines
yaml
jobs:
  - type: requestor
    requests:
      - url: https://example.com/
    tests:
      - name: 'home page was reached'
        type: url
        url: https://example.com/
        operator: or
        responseHeaderRegex: 'Content-Type'
        onFail: error

go deeper

for a junior

Recall that a job can carry tests, that there are four kinds, and that each one needs onFail set to warn, error or info. Know that a test is written under the job it is testing.

for a middle

Explain what each type asserts, that monitor alone is judged during the job and can end it, and how onFail feeds the same error and warning buckets that decide the plan's outcome.

for a senior

Show that you assert on work having happened rather than only on findings, and that you check the run's output for the line confirming each test was actually added rather than trusting the file.

for a principal

Own which assertions belong in the plan and which belong in the pipeline around it. Tests inside the plan travel with it and gate the run; anything needing history or comparison across runs cannot live there.

## Why a plan needs assertions A plan on its own answers "did the run complete", not "did the run do what I meant". Job outcome tests close that gap: under any job's **`tests`** key you attach assertions the framework evaluates for you, so the run itself reports the answer instead of a pipeline step parsing output afterwards. ## The four types | type | what it asserts | when it is judged | |---|---|---| | `stats` | a named statistic compared against a value with an operator | after the job finishes | | `url` | a URL is present or absent, with optional regexes over the request or response | after the job finishes | | `alert` | a finding from a named scan rule is present or absent | after the job finishes | | `monitor` | a named statistic stays under a threshold | **while the job is running** | `monitor` is the odd one out in two ways. It has no operator — it fails when the statistic **exceeds** the threshold — and it is evaluated inside the job's own polling loop, so a failing monitor test **ends that job early**. That makes it the only one that can change the job it is attached to; the other three can only affect what happens once that job has finished. "Stop crawling if authentication failures climb past this" is a monitor test. ## `onFail` is mandatory, and it is the whole verdict Every test carries **`onFail`**, one of **`warn`**, **`error`** or **`info`**. There is no default. A test whose `onFail` is missing or is not one of the three is rejected as it is constructed: the framework records a warning and **drops the test**, so the plan runs with an assertion that silently is not there. If a test matters, spell `onFail` correctly and check that the run's output says the test was added. `onFail` is not decoration, because the bucket it names is the same bucket everything else in the plan writes into: - **`error`** — a failing test can stop the remaining jobs, and gives the run the error exit value. - **`warn`** — the run ends on the warning exit value. - **`info`** — recorded and visible, and it changes neither. So a test with `onFail: info` is documentation; a test with `onFail: error` is a gate. ## Not every test fits every job Support is uneven, and the two mismatches fail in opposite ways — which is the detail most worth carrying: 1. **`url` and `stats` attach to any job.** Anything that can be counted or reached can be asserted on. 2. **`alert` tests attach only to jobs that produce alert data** — the active-scan job, the job that waits for passive scanning to drain, and the sequence active-scan job. Attach one elsewhere and the framework records an **error**, which stops the plan before anything runs. 3. **`monitor` tests attach only to long-running jobs that poll for them** — the crawling jobs and the active-scan jobs. Attach one elsewhere and the framework records a **warning** and silently drops the test; the plan runs on without it. A reader who expects one rule for both will be surprised in one direction or the other: a misplaced alert test is loud and fatal, a misplaced monitor test is quiet and merely absent. ## Statistics have a prerequisite `stats` and `monitor` both read the program's in-memory statistics. If that facility is not available in the build, the framework warns as the tests are added rather than failing them at judgement time — another case where the run tells you in its output that an assertion is not really there. ## Writing tests that are worth having - Assert that work **happened**, not only that nothing was found. A statistic test requiring that some URLs were added is the cheapest defence against the plan that completes having scanned nothing. - Use `alert` tests with `onFail: error` for the findings you have decided a build may not have, and `onFail: info` while you are still measuring how noisy a rule is on your application. - Use a `monitor` test to bound a job that can run away — the threshold ends the job, and the `onFail` level then decides whether ending it early is worth failing over. - Watch the combining operator on a `url` test. With `and`, every one of the request and response regexes it can take has to match, and one you did not write counts as not matching — so an `and` test carrying a single regex can never pass. With `or`, any one match is enough; a `url` test with no regexes at all asserts only that the URL is in the site tree. - Name your tests. The `name` is what appears in the output, and a plan with several unnamed tests on one job is hard to read once one of them fails.

  • What is different about a `monitor` test?
    It is judged while the job runs rather than after it, it has no operator — it fails when the statistic exceeds its threshold — and a failure ends that job early. It is the only one that can change the job it is attached to; the others act only once their job is over.
  • What happens to a test whose `onFail` is missing?
    It is rejected as it is built: the framework records a warning and drops the test, so the plan runs with that assertion silently absent. `onFail` has no default, which is why the run's output should be checked for the line confirming each test was added.
  • Which test would you use to catch a plan that scanned nothing?
    A statistics test on the crawling job, asserting that the count of URLs it added is at or above some floor, with `onFail: error`. A finished run with no findings and a run that never reached the site look identical from the exit value alone.

saying these in an interview costs you the question

  • Thinks onFail has a sensible default
  • Says all four tests are judged after the job finishes
  • Treats a dropped test as if it had passed
  • Attaches an alert test to any job and expects a warning
  • Assumes a passing plan proves the jobs did work