skip to content

Grouping and Labelling

How a screen of red becomes a short list of decisions: rules over the failure text, clustering, machine suggestions and links out to tracker items. Interviewers probe who does the labelling.

on this pageshow

explore

questions

21

In ReportPortal, a launch's generated error clusters are returned as `ClusterInfoResource` objects. What do the `index`, `message`, `matchedTests` and `launchId` fields on one of them tell you?

level: juniorimportance: must knowfreq 58%

answer

  1. one group of similar failures
  2. text, count, identifier, owner
  3. matchedTests is a member count
  4. the scope field names a launch

basics

~20 s

message is the shared failure text the group formed around, matchedTests counts the test items in it, index is the identifier the analyzer gave the group, and launchId scopes the cluster to a single launch.

solid answer

~40 s

ReportPortal's unique-error clustering groups a launch's failing test items by how similar their failure text is, and each group comes back as a `ClusterInfoResource`. Its `message` holds the representative failure text the group formed around -- that text is the whole basis of the grouping. `matchedTests` is the count of test items that landed in the cluster, so it is the first thing to read when checking whether one cluster swallowed the launch. `index` is the identifier the analyzer assigned the group, stored on the entity as `indexId`, while `id` is the cluster row's own database key -- two different numbers. `launchId` is the field people skip: a cluster belongs to exactly one launch, so the same failure family in tomorrow's run is a different cluster row.

go deeper

for a junior

Be able to name the four fields and say what each holds, and state plainly that a cluster belongs to one launch. Knowing matchedTests is a count, not a list, is enough at this level.

for a middle

Explain that the grouping is a text operation over failure messages, that generation is asynchronous and read back separately, and that the resource's index is the entity's indexId under another name.

for a senior

Show you read the matchedTests distribution across the list as a health signal before opening anything, and that you never store a cluster id as a durable key in something downstream.

for a principal

Own the position that clustering is a disposable per-launch view, not a defect registry, and be able to say what you would and would not let a cluster drive automatically.

## What clustering produces ReportPortal's unique-error clustering takes the failing test items of **one launch** and groups them by how similar their failure text is, with nobody having written a rule first. Each group that comes out is one row in the clusters table, and the API renders that row as a `ClusterInfoResource`. Two calls bracket the feature. One **starts** generation for a launch and returns an acknowledgement that generation has started -- the work runs asynchronously in the analyzer service. A second call **reads back** the clusters that now exist for that launch. An empty list a second after the first call means "not finished yet", not "no clusters found". Generation is also refused outright for a launch that is still `IN_PROGRESS`, so the sequence is: finish the launch, start generation, then read. ## The fields, one at a time | field | what it holds | |---|---| | `id` | the cluster row's own database key | | `index` | the identifier the analyzer gave this group; the stored entity holds the same value as `indexId` | | `launchId` | the launch this cluster belongs to | | `message` | the representative failure text the group formed around | | `matchedTests` | how many test items fell into the group | | `metadata` | a free-form map carried alongside the cluster | - **`message` is the entire basis of the grouping.** Clustering here is a text operation over failure messages. Everything a cluster claims is a claim about strings, which is why a message that embeds a request id or a timestamp behaves so differently from one that does not. - **`matchedTests` is a count, not a list.** It is computed as the number of test items linked to that cluster. To see *which* items, you follow the cluster; to judge whether the grouping is any good, the count alone is usually enough. - **`index` and `id` are two different numbers.** `id` is the row's primary key inside ReportPortal. `index` is the identifier the analyzer assigned the group, and the persisted entity calls that field `indexId` -- the API just renames it on the way out. Quoting one where the other was meant is the classic confusion when someone scripts against this endpoint. - **`launchId` is the scope, and it is the field people skip over.** A cluster is a per-launch object. The entity also carries a `projectId`, but that is addressing, not scope: the row still names exactly one launch. ## Reading a cluster list Because `matchedTests` sits on every row, the *shape* of the list is diagnostic before you open a single cluster: 1. One cluster whose `matchedTests` is close to the launch's entire failure count, sitting under a short, generic `message` -- the grouping merged problems that are not the same problem. 2. A long list in which nearly every `matchedTests` is 1 -- the grouping split one problem into many, usually because the messages carry text that varies run to run. 3. A handful of clusters with plausible counts and messages you can read as distinct failures -- the shape you actually want. That reading is the everyday use of the endpoint. You are not looking for a number; you are looking at a distribution. ## What a cluster is not A cluster is a **proposal about text**, not a verdict about defects. Nothing in `ClusterInfoResource` says "these are the same bug", assigns a defect type, or links a ticket. Those are separate acts performed by people and by other parts of the product, on top of a grouping that clustering merely offered. Treating a cluster as a defect record is the single most common overreach, and it shows up as soon as two genuinely different failures land in one cluster because they share a wrapper message. Equally, a cluster is not a rule. Nobody wrote a pattern that produced it, so nobody can point at the pattern to explain why two failures are together -- the only explanation available is the `message` the cluster formed around, plus whatever normalisation was applied before the messages were compared. ## Entity versus resource, and why it matters The stored entity is deliberately narrow: `id`, `indexId`, `projectId`, `launchId`, `message`. It holds **no member list and no count** -- membership lives in the link between items and clusters, and `matchedTests` is derived when the list is read. That is why the count you see is always current for the launch's present state, and why the entity is cheap to rewrite when clustering is run again. So when you read a cluster in an integration, hold on to `launchId` plus `message`, treat `matchedTests` as the health signal, and treat `id` as a handle valid only for as long as this launch's clusters are not regenerated.

  • You start cluster generation for a launch and immediately read the cluster list, which comes back empty. What has happened?
    The start call only acknowledges that generation has begun -- the analyzer does the work asynchronously. An empty list read straight afterwards means generation has not finished, not that nothing grouped. Poll the list, and check that the launch was not still `IN_PROGRESS`, since generation is refused for a launch that has not finished.
  • Why do the stored cluster entity and the API resource use different names for the same value?
    The entity persists the analyzer's group identifier in a field called `indexId`, and the converter copies it onto the resource's `index`. It is a rename on the way out, nothing more. The confusion it causes is real, though: `index` and `id` on the resource are different numbers, and only `id` is the row's database key.
  • The resource carries matchedTests but the stored entity has no count field. Why?
    The entity holds only `id`, `indexId`, `projectId`, `launchId` and `message`; membership lives in the link between test items and the cluster. `matchedTests` is counted when the list is read, so it always reflects the launch's current state and the entity stays cheap to delete and rebuild when clusters are generated again.

saying these in an interview costs you the question

  • Calling a cluster a defect record or a bug
  • Treating index and id as the same number
  • Assuming a cluster spans the whole project
  • Expecting matchedTests to list the member items
open as a page

In ReportPortal, a failure's issue can carry a set of `externalSystemIssues`. What does a single `ExternalSystemIssue` entry hold, and what does attaching one change about that failure?

level: juniorimportance: must knowfreq 62%

basics

~10 s

One externalSystemIssues entry records where a tracker item lives: ticketId, btsUrl, btsProject, a direct url, plus submitDate and pluginName. It carries no ticket status. Attaching one labels the failure without changing its defect type.

open as a page

In ReportPortal, every failed test item's issue carries a defect group from `TestItemIssueGroup`. What does `TO_INVESTIGATE` mean beside `PRODUCT_BUG`, `AUTOMATION_BUG`, `SYSTEM_ISSUE` and `NO_DEFECT`, and which group does a brand-new failure land in?

level: juniorimportance: must knowfreq 66%

basics

~10 s

TO_INVESTIGATE is ReportPortal's undecided defect group, and every new failure is filed there automatically. PRODUCT_BUG, AUTOMATION_BUG, SYSTEM_ISSUE and NO_DEFECT each record a decision already taken, by a person or by the auto-analyzer.

open as a page

In an Allure results directory, what is `categories.json`, and what does one `Category` entry in it hold?

level: juniorimportance: must knowfreq 60%

basics

~20 s

categories.json is a rule file placed in the Allure results directory. Each entry names one bucket and lists the conditions a result must meet to land in it: a message pattern, a trace pattern, matched statuses and a flaky flag.

open as a page

ReportPortal's `CreateClustersRQ` carries only `launchId` and `removeNumbers`. What does `removeNumbers` change about the clusters you get back, and how does each setting fail?

level: middleimportance: must knowfreq 52%

basics

~10 s

removeNumbers strips digits from failure messages before they are compared. Left off, ids and counts split one problem into many clusters; turned on, failures whose only difference was a numeric code merge into one.

open as a page

In ReportPortal, when auto-analysis puts a defect group on a failure it also sets `autoAnalyzed`. What does that boolean record, where does it live, and what does it deliberately not say?

level: middleimportance: must knowfreq 58%

basics

~20 s

autoAnalyzed records provenance — analysis set this defect group, not a person. It is a boolean on the issue rather than on the test item, stored in the auto_analyzed column, and it carries no confidence at all.

open as a page

In Allure 2, a `categories.json` rule whose `messageRegex` is `RuntimeException` catches nothing, while `.*RuntimeException.*` catches the failures you meant — why?

level: middleimportance: must knowfreq 56%

basics

~20 s

Allure matches the whole string rather than searching inside it: the pattern must cover the entire failure message end to end. A bare token matches only a message that is exactly that token, so shipped examples wrap patterns in .* instead.

open as a page

In Allure 3, what can a rule under `resolutions` in `allurerc` match on, and what does a `resolution` of `issue` attach to a failing test?

level: middleimportance: should knowfreq 38%

basics

~20 s

A resolutions rule matches on messageRegexp, testCaseId, retryHash or environment - at least one is required, and every matcher it declares must match. A resolution of issue attaches a tracker id and type, which resolves through resolutions.links into a URL.

open as a page

In ReportPortal, a stored ticket link carries `btsUrl` and `btsProject` rather than the id of a configured integration. What does that pair resolve to when the server needs the tracker, and why must the integration sit in the `BTS` group?

level: middleimportance: should knowfreq 41%

basics

~20 s

The btsUrl and btsProject pair is a lookup key: ReportPortal resolves it to a configured integration, the project's own before a global one, and rejects any integration whose type is not in the BTS group.

open as a page

In ReportPortal's `AnalyzerConfig`, what do `minShouldMatch` and `numberOfLogLines` control, and how does raising each one change the defect-group suggestions a project gets?

level: middleimportance: should knowfreq 50%

basics

~20 s

minShouldMatch is the similarity threshold a new failure must clear against an already-labelled one before that label is reused. numberOfLogLines caps how much of the failure's error log is compared. Raise either and you get fewer, safer suggestions.

open as a page

A ReportPortal cluster row holds `id`, `indexId`, `projectId`, `launchId` and `message`. Given that, how would you follow one recurring failure group across successive launches?

level: seniorimportance: should knowfreq 44%

basics

~20 s

Not through any field the model offers. A cluster row names one launch and holds no pointer to a cluster in another, so the only join available is the message text itself, compared outside the product.

open as a page

A ReportPortal launch's cluster list is useless -- either one cluster holding nearly every failure, or dozens each holding one. How do you tell which it is and what do you actually change?

level: seniorimportance: should knowfreq 41%

basics

~20 s

Read matchedTests and message together across the list. One huge count under short generic text means unrelated problems merged; a page of counts of one means varying text split one problem, and removeNumbers helps only when that text is digits.

open as a page

A ReportPortal project has just been created and its first launches are all red. Auto-analysis is enabled, yet no failure ever gets a defect group. What is happening, and how do you get the project out of it?

level: seniorimportance: should knowfreq 44%

basics

~20 s

A cold start. Auto-analysis reuses defect groups from failures somebody already labelled, and a new project has none, so everything stays in TO_INVESTIGATE. The way out is to label a representative sample by hand, then re-run analysis over the undecided items.

open as a page

In Allure 2, two rules in `categories.json` both match the same failing test — which bucket does it land in, and where does a failure that matches no rule go?

level: seniorimportance: should knowfreq 50%

basics

~20 s

The first matching rule in file order wins, so the array's order is part of the configuration and a broad rule early hides every narrower rule below it. Failed and broken results matching nothing fall into built-in default buckets.

open as a page

For a ReportPortal project, would you let auto-analysis write the defect group straight onto incoming failures, or keep it as a proposal a person accepts — and how do you stop wrong machine labels feeding the next round?

level: principalimportance: should knowfreq 40%

basics

~20 s

Keep it as a proposal a person accepts, and keep the record of who set every label. Applied groups become the evidence for the next round, so a wrong label compounds unless you can find the machine-set ones and reset them.

open as a page

Your team's Allure `categories.json` has collected dozens of rules over two years — how do you decide which rules stay, and who is accountable for pruning it?

level: principalimportance: should knowfreq 40%

basics

~20 s

Treat it as versioned configuration with an owner per rule: a rule earns its place only if the failure recurs, is distinguishable from its message or trace, and has an action attached. Delete dead and shadowed rules at review time.

open as a page

ReportPortal records `rp.cluster.lastRun` after clustering a launch. Where does that key live, and what question does it let you answer?

level: middleimportance: nice to knowfreq 26%

basics

~20 s

It lives as a system attribute on the launch itself, written by the clustering pipeline and replaced on each full generation. It answers whether and when that launch was clustered -- nothing about the clusters produced.

open as a page

In Allure 2's `categories.json`, what do the `matchedStatuses` and `flaky` fields on a rule test, and what can neither of them do?

level: middleimportance: nice to knowfreq 38%

basics

~20 s

Both are match conditions. matchedStatuses lists the statuses a rule accepts; flaky requires the result's own flaky flag to equal the value given. Neither writes anything, so a rule can select a flaky result but never mark one.

open as a page

In ReportPortal, a proposed defect group comes back as a `SuggestInfo` record carrying `matchScore`, `esScore`, `resultPosition`, `esPosition`, `usedLogLines`, `minShouldMatch` and `userChoice`. What is that record for?

level: seniorimportance: nice to knowfreq 30%

basics

~20 s

A SuggestInfo record is the receipt for one proposal: two ranking scores and the positions they gave the candidate, the settings the match ran under, how long it took, which model produced it, and what the person eventually chose.

open as a page