skip to content

When should a new variation in a Playwright suite become another entry in `projects` rather than a `test.use()` in the spec files?

level: principalimportance: should knowfreq 31%

answer

  1. Reach, visibility, cost
  2. A project multiplies the whole run
  3. Only projects are selectable and reported
  4. Narrow a new entry before widening it
  5. Retire columns that catch nothing

basics

~20 s

Make it a project when the variation is an axis the whole suite should run under and be reported on separately. Keep it in test.use when only certain specs need it, since a project re-runs everything.

solid answer

~50 s

The deciding questions are reach, visibility and cost. A `projects` entry re-runs everything it matches, appears in the report as its own dimension, and can be selected with `--project` — so it earns its place when the variation is an axis you want measured across the suite and triaged on its own. `test.use()` in a spec file is the right home when the variation belongs to specific tests: it is invisible in the report and unselectable, which is fine for a local requirement and wrong for an axis. The cost is the other half: N projects over M tests is N×M runs, and each new entry also adds a column of results someone has to look at. Where a project would run almost nothing in common with the rest, narrowing it with `testMatch` is usually better than a whole extra matrix column.

code

typescript · 18 lines
typescript
import { defineConfig, devices } from '@playwright/test';

const chrome = { ...devices['Desktop Chrome'] };

export default defineConfig({
  testDir: './tests',
  retries: 1,
  projects: [
    { name: 'console-chromium', use: chrome },
    { name: 'console-webkit', use: { ...devices['Desktop Safari'] } },
    {
      name: 'console-dashboards',
      testMatch: '**/dashboards/**',
      retries: 2,
      use: chrome,
    },
  ],
});

go deeper

for a junior

Know both places a variation can live: a named entry in the projects array, or a test.use call in the spec file that needs it.

for a middle

Explain the mechanical difference — a project re-runs every file it matches and is named in the report, while a file override changes options wherever that file runs.

for a senior

Argue the cost side concretely: an extra project multiplies wall clock and triage, so narrowing a new entry with testMatch is often the better trade than a full column.

for a principal

Own the matrix as a portfolio — what each column is claimed to measure, how new ones are approved, and how columns that catch nothing are retired before the suite stops meaning anything.

## The three questions that decide it Both mechanisms change option values; they differ in reach, visibility and cost. 1. **Reach** — does the variation apply to the whole suite, or to particular specs? A project re-runs every file it matches under the new options. `test.use()` changes options for one file or one describe block, in every project that runs it. 2. **Visibility** — does the team need to see this dimension in results? Project names appear on every result and can be selected with `--project`. A file-level override appears nowhere; the report still shows the test under each project name, just not varying as the name implies. 3. **Cost** — a project multiplies. Adding an entry to a matrix of M tests adds M runs of wall clock, M more results to triage, and one more column that can go red for reasons unrelated to the change under review. ## A decision table | Signal | Make it a project | Keep it in `test.use()` | |---|---|---| | Applies to the whole suite | Yes | No | | You want to run it alone in CI | Yes, via `--project` | Not addressable | | You want failures attributed to it | Yes, by project name | No separate attribution | | Only a handful of specs care | Usually not worth a column | Yes | | Doubling wall-clock is acceptable | Required | Not incurred | ## Middle grounds worth knowing The choice is not binary, and the interesting answers usually live between the two: - **A narrowed project.** An entry with its own `testMatch` or `testDir` runs only the slice that benefits from the variation, keeping the axis visible and selectable without multiplying the whole suite. This is often the right answer when a full column would be mostly redundant. - **A project plus a small exclusion.** Run the axis broadly but drop the few specs it cannot support with that project's `testIgnore`, rather than abandoning the axis. - **A separate config file** run with `--config`. Justified when the variation is really a different suite with its own lifecycle and its own report — a nightly deep run, say — not a dimension of the same run. The cost is that it will drift, since nobody sees it in the default invocation. ## The failure modes at each extreme Too many projects: - CI wall-clock grows multiplicatively while defect yield does not; the fourth column finds almost nothing the first three missed. - Triage cost rises faster than run time, because every failure now has to be classified by variant before anyone can act on it. - Columns nobody looks at get their retries raised until they are green by construction. Too much in `test.use()`: - Variations hide inside spec files where no one reviews them as a set. - A file that pins an option is still paid for in every project while no longer varying along that axis — cost without coverage. - Nothing can be run or reported in isolation, so "does the suite still pass under X" has no answer. ## Governing the matrix over time 1. Require a stated reason for every project entry: what does this column tell us that no other column does? 2. Prefer narrowing a new entry with `testMatch` first; widen it only if the narrow version keeps finding things. 3. Review the matrix on a schedule, and retire an entry that has not caught a distinct failure — removing a column is a legitimate result, not an admission of failure. 4. Keep option variation that is *not* an axis in spec files, and keep it visible by convention: `test.use()` at the top of the file, never buried mid-file. 5. Treat any project whose retries have been raised as on notice; it is a column that is no longer reporting honestly. The underlying principle: a `projects` entry is a **claim that a dimension deserves independent measurement**, and it is paid for in wall clock and triage on every single run. Make the claim deliberately, and let everything narrower live in the specs.

  • How would you keep a Playwright matrix from growing without bound as new variations are proposed?
    Require each entry to justify what it measures that no other entry does, default new ones to a `testMatch`-narrowed slice, and review the matrix on a schedule so entries that have caught nothing distinct are retired. Growth is easy to approve one entry at a time; the review is what makes the cost visible.
  • When is a separate config file the better answer than another project?
    When the variation is a different suite rather than a dimension of the same one — a different cadence, a different owner, a report nobody reads alongside the main run. Keep it separate then, but expect drift: anything outside the default invocation stops being maintained.
  • What signals that an existing project has stopped earning its place?
    It fails only for reasons already caught elsewhere, its retry budget has been raised to keep it green, or nobody can name a defect it found. Any of those means the column costs wall clock and triage while contributing no independent signal, and removing it is the honest move.

saying these in an interview costs you the question

  • Adds a project for a variation only two specs need
  • Assumes another project costs only extra machine time
  • Never removes a project once it exists
  • Treats test.use as a substitute for a reported axis
  • Judges matrix size by column count rather than signal