One project in a Playwright matrix is intermittently red while the others are green — which project-level options contain it without editing the specs?

## Diagnose in the failing variant first A matrix failure is only useful once you know it is one variant and not the suite. The first moves: 1. `npx playwright test --project=admin-webkit` — run the suspect variant alone and see whether it reproduces at all. 2. Run it again a few times; a failure that comes and goes is a different problem from one that fails every time in that engine and passes in the others. 3. Compare with a green variant on the same spec, so you can tell "this engine behaves differently" from "this test is racy everywhere and this engine is just slower". The project name in the report is the handle for all of this: the reporter prints it next to every result, so a failure already tells you which variant to select. ## Contain it at the narrowest scope that works Once you know which variant is affected, the config gives you three graded levers, all scoped to a single entry of `projects`: - **`retries` on that project** overrides the top-level `retries` for it alone. The rest of the matrix keeps whatever budget it had, so genuine flakiness elsewhere still surfaces. - **`use` on that project** carries option differences the variant needs, rather than moving them to the config root where every project inherits them. - **`testIgnore` on that project** removes the specs the variant genuinely cannot run, leaving them running everywhere else. ```ts { name: 'admin-webkit', retries: 2, testIgnore: '**/csv-export.spec.ts', use: { ...devices['Desktop Safari'] }, } ``` Each of these is a line in the config that a reviewer sees in a diff. That visibility is the point: it is an explicit statement about one variant, not a change to how the whole suite behaves. ## What not to reach for | Tempting fix | Why it backfires | |---|---| | Raise the top-level `retries` | Masks flakiness in every project, not just the failing one | | `test.use()` in the spec file | Applies in every project running that file, flattening the axis | | Delete the spec | Loses the coverage in the projects where it was passing | | Drop the project from the matrix | Throws away the whole variant to silence one spec | The common thread: every one of these is a suite-wide change bought to fix a one-variant problem. The project entry exists precisely so the fix can be as narrow as the problem. ## Retries are containment, not a cure Raising `retries` on a project stops one shaky engine from blocking the pipeline today; it does not explain anything. Treat it as a deliberate, time-boxed decision: - Record why the project's budget differs from the rest — a comment on the entry is enough. - Watch the flaky count for that project; a test that only ever passes on retry is still broken. - Revisit when the underlying cause is fixed and put the budget back. A project quietly sitting at a higher retry count for a year is how a matrix stops meaning anything: its results look green while the variant it names has not truly passed in months. ## A checklist for the situation 1. Reproduce with `--project` on the failing variant. 2. Decide whether the behaviour is a real engine difference or an ordinary race. 3. If it is a real difference, express it in that project's `use`. 4. If the variant cannot support the spec at all, exclude it there with `testIgnore` and say so. 5. Only then, if the pipeline needs breathing room, raise `retries` on that project alone — and leave a note about when it comes back down.