skip to content

In an Allure 2 report, `widgets/suites.json` does not list every suite in the run. What does it list, in what order, and what is the `total` that sits beside those rows?

level: seniorimportance: nice to knowfreq 32%

answer

  1. a leaderboard, not an inventory
  2. top-level groups only, ranked by failures
  3. the row count and the total differ
  4. statistics counted over leaves, not children

basics

~20 s

It lists at most ten rows, built only from the top layer of the suite tree and ranked failures-first. The total beside them counts every top-level group in the whole tree, so the panel is a worst-ten board rather than an inventory.

solid answer

~40 s

Allure 2's suites plugin writes the same name into two directories: `data/suites.json` is the full tree the Suites tab renders, and `widgets/suites.json` is the overview panel. The widget form is a `total` plus an `items` array of `uid`, `name` and `statistic`. Only groups sitting directly under the tree's root become candidate rows; they are sorted by the statistic descending — most `failed` first, then `broken`, then `passed`, then `skipped`, then `unknown` — and cut to ten. `total` is the count of top-level groups in the whole tree, not the row count and not a test count, so a report can honestly show ten rows beside a much larger total. Each row's statistic is computed over the leaves beneath the group, so it counts individual results however deep they sit.

code

json · 21 lines
json
{
  "total": 87,
  "items": [
    {
      "uid": "6f1b0c3d9a2e4f5b8c7d1e0a3b4c5d6e",
      "name": "checkout",
      "statistic": {
        "failed": 6, "broken": 0, "skipped": 1,
        "passed": 40, "unknown": 0, "total": 47
      }
    },
    {
      "uid": "1a2b3c4d5e6f708192a3b4c5d6e7f809",
      "name": "search",
      "statistic": {
        "failed": 0, "broken": 2, "skipped": 0,
        "passed": 118, "unknown": 0, "total": 120
      }
    }
  ]
}

go deeper

for a junior

Know that this panel shows a shortened list rather than every suite, and that the number beside the rows is not the row count. Check the full tab before saying anything about a suite you cannot see on the overview.

for a middle

Explain the three reductions in order: only the tree's top layer becomes candidates, the candidates are sorted by the statistic descending, and the list is cut to ten. Say what the total counts and what each row's statistic is summed over.

for a senior

Demonstrate that you would not accept the panel as evidence about a suite missing from it, and that you spot the tie-break putting a large clean suite above a small one. Reach for the full tree file whenever the question needs a census.

for a principal

Own the difference between a triage surface and a record. Decide what teams may conclude from a capped, failure-ranked panel, and where a question about coverage across all suites has to be answered from a queryable source instead.

## The panel is a projection, not an inventory Allure 2's suites plugin writes the file name `suites.json` twice into the generated report, under two directories: | path | what it is | |---|---| | `data/suites.json` | the whole suite tree the Suites tab renders | | `widgets/suites.json` | the trimmed panel the overview draws | Only the second is a widget. It is a small object with two keys — a `total` and an `items` array — and each entry of `items` carries a `uid`, a `name` and a `statistic`. Everything about how it is built is subtractive: it starts from the same tree as the tab and throws most of it away. ## What gets kept, and in what order Three reductions happen in sequence, and each is easy to forget when reading the panel: 1. **Only the top layer of the tree becomes rows.** The candidates are the groups sitting directly under the root, not every nested group. Which groups sit at that layer depends on the suite labels the run actually wrote, so two runs of the same tests can produce different rows without anything about the panel changing. 2. **Rows are ranked failure-first.** The comparison walks the statistic in a fixed order — `failed`, then `broken`, then `passed`, then `skipped`, then `unknown` — and sorts by it descending, so the group with the most failures leads. Ties fall through to the next field, which has a consequence people rarely predict: among groups with no failures and none broken, the one with the most **passes** takes the higher slot. The order is neither alphabetical nor size-neutral. 3. **The list is cut to ten rows.** However broad the tree really is, the panel shows at most ten. ## What `total` counts `total` is the number of top-level groups in the **whole** tree — not the number of rows shown, and not a number of tests. A report with eighty-seven top-level suites shows ten rows beside a `total` of eighty-seven. That field is what keeps the panel honest: it is the file telling you, in writing, that you are looking at a sample. Each row's `statistic` is computed recursively over the **leaves** beneath that group, so it counts individual results however deep they sit, not immediate children. A top-level group with four sub-suites under it reports the results in all of them, not the number four. ## How to read it without being misled - **Absence is not a pass.** A suite missing from the panel may simply be the eleventh worst. The only way to know is `data/suites.json`, or the Suites tab that renders it. - **A short panel is a small tree, not a filter.** Fewer than ten rows means the tree genuinely has fewer than ten top-level groups. - **Rank is not severity.** One failure in a huge suite outranks a small suite that is entirely green, and is itself outranked by any group with two failures, regardless of how many tests each ran. The ranking knows nothing about proportion. - **`total` is the sanity check.** When it is much larger than the row count, say so out loud while quoting the panel: "the ten worst of eighty-seven suites", never "our suites". ## Why the design is defensible anyway An overview panel exists to answer "where do I click first", and failure-first ranking with a hard cap answers exactly that: the groups most worth opening are at the top, and the panel stays a fixed size however large the run gets. The cost is that it cannot be used as a census, and the `total` beside the rows is the file admitting as much. Treat the widget as a triage board and the tab behind it as the record, and both do their jobs. Treat the widget as the list of your suites and you will report on ten of them for as long as nobody checks — which, since the panel looks complete, can be a very long time.

  • Where do you go for the suites the panel left out?
    `data/suites.json` in the same generated report — the full tree the Suites tab renders, written by the same plugin under the same name in a different directory. Nothing is lost by the trimming; the panel is a projection of that tree, so the complete list is one file away.
  • Two suites both have zero failures and none broken. Which appears higher?
    The one with more passes. The ranking walks the statistic in order — failed, broken, passed, skipped, unknown — descending, so once the first two fields tie it falls through to the pass count. Among clean groups the largest takes the slot; the order is neither alphabetical nor size-neutral.

saying these in an interview costs you the question

  • Reads the panel as the complete list of suites
  • Treats absence from the panel as a passing suite
  • Thinks total counts tests rather than top-level groups
  • Assumes rows are ordered alphabetically or by size