Marking a Flaky Outcome

## Two separate things on one result Allure's writer model, the one `allure-java` serialises into a results directory, keeps a test's **outcome** and a test's **reputation** in two different places. The flaky mark belongs to the second. The outcome is the result's `status` field, typed by `io.qameta.allure.model.Status`. That enum declares exactly four constants: `FAILED`, `BROKEN`, `PASSED` and `SKIPPED`. There is no fifth constant for flakiness, no `FLAKY` member, and no way to smuggle one in, because the serialiser writes the enum's own lowercase wire value. A `-result.json` can therefore only ever say `"status": "failed"`, `"broken"`, `"passed"` or `"skipped"`. The reputation lives on `StatusDetails`, an optional object hanging off the result. `StatusDetails` declares seven fields, in this order: 1. `known` -- a boolean. 2. `muted` -- a boolean. 3. `flaky` -- a boolean. 4. `message` -- the short failure text. 5. `trace` -- the stack trace. 6. `actual` -- the observed value. 7. `expected` -- the value the assertion wanted. The flaky mark is item three: a plain boolean, flipped with `setFlaky(...)`. ## What setFlaky(...) actually changes Nothing except that one boolean. - It does **not** change `status`. A result marked flaky keeps whatever outcome the run gave it. - It does **not** turn a red run green, and it does not make a green run suspect to the runner. - It does **not** record anything about attempts. The flag is a claim about the case, not a log of what this execution did. - It does **not** fill `message` or `trace`; those still carry the failure text, if there was one. Because the two are orthogonal, all four combinations are legal and all four occur in real results directories: a `passed` result with `flaky` false (the ordinary case), a `passed` result with `flaky` true (a case somebody declared unreliable that happened to work), a `failed` result with `flaky` false, and a `failed` result with `flaky` true. ## What it looks like on disk ```json { "name": "checkout completes with a saved card", "status": "failed", "statusDetails": { "known": false, "muted": false, "flaky": true, "message": "expected the order id to be present" } } ``` `"status"` is untouched by the flag sitting beside it. Note also that all three booleans are written even though two of them are `false`. They are primitive booleans on the model class, and the writer's mapper omits only nulls, so a primitive can never be dropped. Absence of the whole `"statusDetails"` object is possible, and happens when nothing set any of its fields, but absence of one boolean while its siblings are present is not. ## Why a flag and not a status | | `status` | `statusDetails.flaky` | |---|---|---| | type | four-valued enum | boolean | | always present | yes | only when `statusDetails` exists | | what it records | how this execution ended | a claim about the case | | who must handle it | every reader | only readers that care | Modelling flakiness as a fifth status would force every consumer of the four to grow a branch for it, and would make the two facts mutually exclusive: a case could then be flaky or failed but not both, which is exactly the pair you most want to see together. A side flag is ignorable by construction. A tool that knows nothing about flakiness reads `status`, gets a well-formed answer, and never notices the boolean. ## What this means when you read results Anything that counts flaky cases has to read the boolean. A dashboard, a filter or a gate built by switching on `status` alone will report zero flaky tests no matter how many carry the mark, because the mark is not in the field it is looking at. Equally, a case that is genuinely marked is still counted in whatever bucket its `status` puts it in: marking a failing test flaky does not remove it from the failure count, and it does not move it out of `FAILED` into some softer category. The same shape holds for both live Allure majors, because both read the file `allure-java` writes. The flag is a boolean parsed off the result and the status stays one of a small closed set. Where the majors differ is in what they do with the flag afterwards, not in where it lives.