Allure 3's `quality-gate` command validates rules as results are read, and `--fast-fail` makes it exit at the first breach. Why can a suite that retries its failed tests turn that gate red on a run whose finished report is green?

## The command validates while it reads It is tempting to picture a gate as something that happens once, at the end, over a finished set of results. Allure 3's `quality-gate` command is not built that way. It starts the report, subscribes to results as the reader emits them, and validates each incoming batch against state that accumulates across batches. A failure-counting rule does not recount from scratch on every batch; it adds this batch's failures to the running total it stored last time and compares the total against the threshold. That design is what makes an early exit possible at all. If the total has already passed the ceiling, no later batch can bring it back down, so there is nothing to gain by reading the rest. `--fast-fail` turns that observation into behaviour: at the first breach the command prints what it found and exits non-zero immediately. There is also a second, ordinary pass. When the reader has finished, the command validates once more over **all** results, this time with retries explicitly excluded, and that pass produces the verdict when nothing fast-failed. ## How a retried test is recognised Allure 3 does not read "this is attempt two" out of a result file. The reader computes a grouping key -- `retryHash` -- from the test case's identity, a hash of its parameters, and the environment it ran in. Results that produce the same key are attempts at the same thing. The store keeps those attempts in a list and, **every time another attempt is added, re-sorts the list and recomputes the flags**: the attempt that sorts first is the surviving one and is not a retry; every other member of the group is flagged as a retry. Validation then skips anything flagged as a retry, along with anything a known-issues rule has resolved as muted or accepted, and anything already processed in an earlier batch. The consequence is the whole of this question. **Being a retry is not a property a result has when it arrives. It is a property the store confers on it later, when a sibling attempt shows up.** ## The sequence that produces the false red 1. The suite runs a test, it fails, and the framework retries it. Two results are written for that test. 2. The reader reaches the failing attempt first. Its group has one member, so it sorts first, so it is not a retry. 3. The gate validates that batch. The failure counts. If the ceiling was zero, the rule breaches. 4. `--fast-fail` is set, so the command prints the breach and exits non-zero. 5. The passing attempt is still sitting in the results directory, unread. Had it been read, the earlier attempt would have been re-flagged as a retry and dropped from validation, and the final pass would have found no failure at all. Run the same directory without `--fast-fail` and the outcome inverts: the early breach is recorded but stops nothing, the reader finishes, the final pass runs over all results with retries excluded, and the gate goes green. **So the verdict depends on the order the results were read**, which depends on the file system, which is not something you control or can usefully reason about. That is not a bug you can configure away; it is what a decision taken before all the evidence has arrived means. ## What to do about it - **Do not combine an early-exit gate with framework-level retries.** If your suite retries, run the gate over the finished results directory and accept that you pay for the whole read. The point of retrying is that the first outcome is not the answer, and a gate that acts on the first outcome has contradicted that. - **Note that Allure's own runner takes the same position.** Allure 3's `run` command declines to validate the gate at all when it is asked to rerun failures: it runs the test command and warns that gate validation was skipped for that run. The tool has already concluded that these two features do not compose. - **If you genuinely need the fast exit, attach it to a rule retries cannot affect.** The stop-at-first-breach behaviour belongs to a rule group rather than to the gate as a whole, so a group holding, say, an environment-coverage rule can end the command early while the failure-count and success-rate rules sit in a group that is only reported at the end. - **Be precise about what you are buying.** The fast exit shortens the tail of the *reading* step, not the test run -- the tests have already executed and written their results. On a very large results directory that is worth something; on a small one it is noise beside the risk. ## The general shape of the trap Any gate that evaluates a stream, over a corpus where a later record can retract an earlier one, will sometimes act on a record that was about to be retracted. The retry is the everyday instance, but the same reasoning covers any late-arriving result: a shard that finishes last, an attachment that lands after its result, an environment that only appears once its first test has been read. If a rule's input can still change, an early verdict on it is a guess.