skip to content

In Allure 2's `categories.json`, what do the `matchedStatuses` and `flaky` fields on a rule test, and what can neither of them do?

level: middleimportance: nice to knowfreq 38%

answer

  1. conditions, not instructions
  2. the flag is read, never written
  3. statuses are lowercase strings
  4. absent means no restriction at all
  5. one more status than an adaptor can write

basics

~20 s

Both are match conditions. matchedStatuses lists the statuses a rule accepts; flaky requires the result's own flaky flag to equal the value given. Neither writes anything, so a rule can select a flaky result but never mark one.

solid answer

~40 s

They are the two non-regex conditions on a `Category`, and both are predicates. `matchedStatuses` is a list of lowercase status names — the report-side vocabulary is `failed`, `broken`, `passed`, `skipped` and `unknown` — and an empty or absent list accepts any status. `flaky` is a tri-state: leave it unset and the rule is indifferent to the flag; set it and the result's own flaky flag must equal the value given. The important part is what neither can do: **a rule never writes**. It cannot mark a result flaky, cannot change a status, cannot mute anything. That is also why a bucket defined with `"flaky": true` stays empty unless something upstream in the pipeline actually put that flag on the results before the report was generated.

code

json · 7 lines
json
{
  "name": "Known flaky timeouts",
  "description": "Only collects results already flagged flaky",
  "matchedStatuses": ["failed", "broken"],
  "flaky": true,
  "messageRegex": ".*Timeout.*"
}

go deeper

for a junior

Remember that matchedStatuses takes lowercase status names, and that a condition you leave out of a rule places no restriction rather than blocking everything.

for a middle

Explain that flaky on a rule is read and not written, and that its unset state makes the rule indifferent to the flag while true and false each narrow it in opposite directions.

for a senior

Be able to explain a stubbornly empty bucket: a rule requiring the flaky flag collects nothing unless something upstream is genuinely putting that flag on results before the report is generated.

for a principal

Decide whether failure buckets should depend at all on a flag produced elsewhere in the pipeline, and name who guarantees that dependency still holds after the suite or its adaptor changes.

## The two conditions that are not regular expressions A `Category` in `categories.json` carries four match conditions. Two of them, `messageRegex` and `traceRegex`, test text. The other two test structured values on the result, and those are the ones people misread. All four combine with **AND**: a rule matches only when every condition it declares holds. A condition left out of the rule places no restriction at all. That asymmetry matters here, because both of these fields have a meaningful "absent" state that is not the same as a false value. ## `matchedStatuses` - It is a **list of lowercase status strings**, serialised exactly that way in the file. - The report-side vocabulary has **five** members: `failed`, `broken`, `passed`, `skipped`, `unknown`. - **An empty or absent list accepts any status.** This is the trap running in the opposite direction from the flaky field: people write `"matchedStatuses": []` intending "none" and get "all". - A rule may legitimately name several statuses, and a bucket for infrastructure trouble usually wants both `broken` and `unknown`. There is a small asymmetry worth knowing. The writer-side model an adaptor uses, `io.qameta.allure.model.Status`, has only four values: `FAILED`, `BROKEN`, `PASSED`, `SKIPPED`. The reader side has a fifth, `unknown`. So `categories.json` will happily accept a status string that no adaptor can ever write onto a result, and a rule requiring only `unknown` may simply never fire against adaptor-produced results. The file is written against the reader's vocabulary, not the writer's. ## `flaky` — a predicate, not a setter This is the field the question really turns on. `flaky` on a `Category` is a **fifth condition beside status, message and trace**, and it behaves as a tri-state: | value in the file | effect on matching | |---|---| | absent / null | the rule does not care about the flag; flagged and unflagged results are both eligible | | `true` | only results whose own flaky flag is set are eligible | | `false` | only results whose own flaky flag is not set are eligible | **A rule never marks anything flaky.** It reads a flag that is already on the result by the time the report is generated — written into the result data further upstream — and uses it as one more filter. Nothing in this file can create, clear or infer that flag. The consequence in practice is a bucket that stays stubbornly empty. Someone writes a rule called "Known flaky timeouts" with `"flaky": true`, expects it to collect the intermittent failures, and gets nothing — because no part of the pipeline is actually setting the flag on those results. The rule is correct; the precondition it depends on was never established. Diagnosing that means looking at the results, not at the rule. ## Combining the two A few shapes come up repeatedly: 1. **Status only.** `{"name": "Environment", "matchedStatuses": ["broken", "unknown"]}` — a catch-all for anything the run could not classify. With no text condition it matches broadly, so its position in the file matters a great deal. 2. **Status plus message.** The everyday shape: narrow the status, then name a distinctive fragment of the message. 3. **Flag plus status.** `{"flaky": true, "matchedStatuses": ["failed"]}` — only meaningful if the flag is genuinely being set upstream. 4. **Flag negated.** `"flaky": false` deliberately excludes results already flagged, which is one way to stop a "needs attention" bucket swallowing everything. ## Why interviewers like this one It separates people who have read the field list from people who have reasoned about it. The name `flaky` reads like an instruction — *make this flaky* — and a candidate who has only skimmed the documentation will say the rule marks results. The correct model is that **`categories.json` is a query over results, not a mutation of them**. Everything in the file selects; nothing in it writes. Once that is clear, the empty-bucket symptom, the accept-anything behaviour of an absent condition, and the presence of a status the writer cannot emit all follow from the same idea. The same reasoning tells you where to spend effort: the file is only as good as the data underneath it, so before adding conditions that depend on flags or on unusual statuses, confirm that something is actually producing them.

  • Someone writes `"matchedStatuses": []` meaning the rule should match nothing. What actually happens?
    It matches every status. An empty list is treated as no status restriction at all, so the rule falls back to whatever other conditions it declares — and if it declares none, it accepts every result it is offered. If you want a rule to be inert, delete it rather than trying to write an impossible condition.
  • A rule with `"flaky": true` never collects anything. Where do you look first?
    At the results, not at the rule. The condition tests a flag that must already be on the result data before generation, so an empty bucket usually means nothing upstream is setting it. Confirm the flag is present on at least one result before spending any time on the pattern or the status list.

saying these in an interview costs you the question

  • Says a rule with flaky true marks results as flaky
  • Thinks an empty matchedStatuses list matches nothing
  • Writes statuses in upper case and expects them to match
  • Assumes the file can change a result's status or mute it