skip to content

In Allure 2, two rules in `categories.json` both match the same failing test — which bucket does it land in, and where does a failure that matches no rule go?

level: seniorimportance: should knowfreq 50%

answer

  1. order in the array matters
  2. first match, not best match
  3. broad rules shadow narrow ones
  4. a default bucket, but not for all

basics

~20 s

The first matching rule in file order wins, so the array's order is part of the configuration and a broad rule early hides every narrower rule below it. Failed and broken results matching nothing fall into built-in default buckets.

solid answer

~50 s

Rules are evaluated in array order and the **first** one that matches takes the result — not the most specific, and not all of them. So the order of `categories.json` is configuration, and a broad rule placed above a narrow one leaves the narrow one permanently dead with no warning anywhere. Put narrow rules first and any deliberate catch-all last. A failure matching nothing is not dropped: in Allure 2 a `failed` result falls into the built-in bucket rendered *Product defects*, and a `broken` one into *Test defects*. Note the edge — those defaults cover failed and broken only, so an unmatched `skipped` or `unknown` result gets no bucket at all. The way to verify any of this is to regenerate from a kept results directory with the order changed, since generation reads results already on disk.

code

json · 11 lines
json
[
  {
    "name": "Checkout timeouts",
    "matchedStatuses": ["failed"],
    "messageRegex": ".*Timeout.*"
  },
  {
    "name": "Everything else that failed",
    "matchedStatuses": ["failed"]
  }
]

go deeper

for a junior

Know that the rules are evaluated in the order they appear and that the first match takes the result, so the order of the array is not cosmetic.

for a middle

Explain both failure shapes: a broad rule above a narrow one leaves the narrow one permanently dead, and a failure matching nothing still appears, in the built-in default buckets.

for a senior

Show a diagnosis path — keep the results directory, move or remove one rule, regenerate, compare bucket membership — rather than reasoning about the patterns on paper.

for a principal

Set the convention and enforce it: narrow rules above broad ones, at most one deliberate catch-all, and reordering treated as a reviewable change to shared configuration.

## First match, not best match Allure 2 walks the `categories.json` array in order, tests each rule against the result, and stops at the first one that matches. That result then belongs to exactly one bucket. Three consequences follow immediately: - **Order is configuration.** Reordering the array changes the report without changing a single pattern. - **Specificity is irrelevant.** A rule matching on a precise exception type does not beat a rule matching on `.*` above it; position decides. - **A result lands in one bucket, not several.** A failure that plausibly belongs to two categories is filed under whichever rule came first, so overlapping rules are a decision you make by ordering rather than something the tool resolves for you. ## The two order bugs, and how each looks **The shadowed rule.** A broad rule — typically one with only `matchedStatuses` and no text condition, or a `messageRegex` of `.*` — sits above narrower rules. Every result it accepts is consumed there, so the narrow rules below never fire. The symptom is one huge bucket and several empty ones, and nothing in the build output mentions it. **The rule that never fires anyway.** A rule further down is dead not because something shadows it but because its own pattern is wrong. This is the cruel part: **both bugs present identically** — an empty bucket — and so does a third, entirely innocent explanation, namely that the failure mode did not occur on this run. Reading the file cannot tell them apart. ## Where an unmatched failure goes A failure matching no rule is not silently dropped. Allure 2 assigns built-in defaults, rendered in the report as *Product defects* for a `failed` result and *Test defects* for a `broken` one. Allure 3 also reads a `categories.json` from the results directory, and additionally accepts rules from its own config file; its default buckets are named *Product errors* and *Test errors*. The edge worth knowing: **the defaults cover failed and broken results only.** A result with any other status that matches no rule gets no bucket at all, which is why a rule intended to collect, say, skipped results has to exist explicitly — no default will catch them. ## Ordering discipline 1. **Narrow rules first**, broad rules last. Sort by how much each rule could possibly accept, most specific at the top. 2. **At most one deliberate catch-all**, at the very bottom, where its job is to be the thing that fires when nothing else did. 3. **Never put a status-only rule near the top.** A rule carrying just `"matchedStatuses": ["failed"]` accepts every failed result in the run. 4. **Treat a reordering as a real change.** It is reviewable, it changes what the report says, and it deserves the same scrutiny as editing a pattern. ## Diagnosing it for real Because rules are applied at generation time over results already on disk, the whole investigation is a loop that runs in seconds with no test execution: 1. **Keep a results directory** from a run containing the failures you care about — the same directory can be generated any number of times. 2. **Generate once with the file as-is** and note which buckets are empty. 3. **Move the suspect rule to the top of the array** and generate again. If it now fills, it was shadowed and the fix is ordering. If it is still empty, the rule itself is the problem — most often a pattern written as a search fragment rather than as a whole-string match. 4. **Remove the broad rule entirely** and generate a third time to see what it had been absorbing. That list is usually the most informative output of the exercise, because it shows the failure modes the catch-all was hiding. 5. **Put it back, correctly ordered**, and confirm the buckets you expect. ## The operational risk The reason this is a senior question rather than trivia is what a badly ordered file does over time. A catch-all near the top turns the category view into a single mound of red nobody reads, and — worse — it absorbs *new* failure modes as they appear. The whole point of bucketing is that an unfamiliar failure stands out; a greedy early rule guarantees it does not. The file degrades quietly, in a way no test and no build step will ever report, which is exactly why ordering belongs in review rather than in one person's head.

  • Can one failure appear in two buckets if two rules match it?
    No. Evaluation stops at the first matching rule, so each result belongs to exactly one bucket. If a failure genuinely spans two concerns you have to choose which rule sits higher, or split the patterns so they no longer overlap. There is no multi-bucket membership to fall back on.
  • An unmatched result that is neither failed nor broken — what does the category view show for it?
    Nothing. The built-in defaults cover failed and broken results only, so an unmatched result with another status simply does not appear in the category view. If you want skipped results grouped, write an explicit rule naming that status in its condition; no default will catch them for you.

The array behaves like a mail filter or a firewall rule list: the first rule that matches decides, so a wide rule at the top quietly makes every rule beneath it dead — and nothing tells you they are dead.

saying these in an interview costs you the question

  • Thinks the most specific matching rule wins
  • Believes a failure can appear in several buckets at once
  • Puts a status-only catch-all at the top of the file
  • Assumes an unmatched failure disappears from the report