skip to content

With PHPUnit 13, what makes a test risky, and why does a run with risky tests still exit with code 0 unless failOnRisky is set?

level: middleimportance: should knowfreq 40%

answer

  1. passed, but something looks wrong
  2. no assertions is risky by default
  3. OK, but there were issues!
  4. failOnRisky and failOnWarning default false
  5. failOnPhpunitWarning defaults to true

basics

~20 s

A risky test passed but broke a rule PHPUnit checks, such as performing no assertions or printing output. Risky results are reported but do not fail the run by default, because failOnRisky defaults to false.

solid answer

~40 s

**Risky** is PHPUnit's label for a test that did not fail yet looks suspect. The default check is `beStrictAboutTestsThatDoNotTestAnything` (on): a test with no assertions is risky. Opt-in checks add more: `beStrictAboutOutputDuringTests` (unexpected output), `beStrictAboutChangesToGlobalState` (with globals backup on), and the coverage-metadata settings. PHPUnit prints them and ends with "OK, but there were issues!", but the exit code stays 0 because `failOnRisky` defaults to false; the same is true of PHP warnings under `failOnWarning`. PHPUnit's own warnings - misconfiguration, a bad data provider, a missing coverage driver - do fail the run, since `failOnPhpunitWarning` defaults to true. In CI I set `failOnRisky` and `failOnWarning` to true, or `failOnAllIssues`, so problems break the build.

code

xml · 13 lines
xml
<phpunit xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:noNamespaceSchemaLocation="vendor/phpunit/phpunit/phpunit.xsd"
         bootstrap="vendor/autoload.php"
         beStrictAboutOutputDuringTests="true"
         failOnRisky="true"
         failOnWarning="true"
         displayDetailsOnAllIssues="true">
    <testsuites>
        <testsuite name="unit">
            <directory>tests/Unit</directory>
        </testsuite>
    </testsuites>
</phpunit>

go deeper

for a junior

Recall that a test with no assertions is reported as risky, and that risky tests do not fail the run unless configured to.

for a middle

Explain each risky check and its default, the exit codes, and the difference between failOnWarning and failOnPhpunitWarning.

for a senior

Show how you tighten a legacy suite step by step: read the issues, fix them, then turn on failOnRisky, failOnWarning or failOnAllIssues in CI.

for a principal

Frame the failOn* policy as a quality gate: what the team may ignore, what must break the build, and how to keep the output readable.

## Outcomes beyond pass and fail PHPUnit 13 reports more than passed and failed. A test can **error** (an unexpected exception), be **skipped** or **incomplete**, or be **risky**: it did not fail, but it broke a rule PHPUnit was asked to check. Separately, tests can **trigger issues** - PHP deprecations, notices and warnings - and PHPUnit itself can emit **PHPUnit warnings** about its own configuration or your test code's metadata. ## What makes a test risky | Check (root attribute) | Default | Risky message | |---|---|---| | `beStrictAboutTestsThatDoNotTestAnything` | on | "This test did not perform any assertions" | | `beStrictAboutOutputDuringTests` | off | "Test code or tested code printed unexpected output" | | `beStrictAboutChangesToGlobalState` | off | "This test modified global state but was not expected to do so" | | `beStrictAboutCoverageMetadata` | off | "This test executed code that is not listed as code to be covered or used" | | `requireCoverageMetadata` | off | "This test does not define a code coverage target but is expected to do so" | | `enforceTimeLimit` with size limits | off | "This test was aborted after N seconds" | Two notes. The global-state check only has something to compare when `backupGlobals` or `backupStaticProperties` is enabled, because PHPUnit snapshots state only then. And a test marked with `#[DoesNotPerformAssertions]` that does assert something is risky too. ## Why risky does not fail the run The exit code is computed from a set of `failOn*` switches, each a root attribute of `phpunit.xml` with a CLI twin: - A run with errors exits with code **2**; with failures, code **1**; otherwise **0**. - `failOnRisky`, `failOnWarning`, `failOnNotice`, `failOnDeprecation`, `failOnSkipped`, `failOnIncomplete` and `failOnEmptyTestSuite` all **default to false**. When only these occur, PHPUnit prints "OK, but there were issues!" and exits 0. - `failOnPhpunitWarning` **defaults to true**. Warnings PHPUnit raises about the run itself - an invalid configuration, a data-set argument-count mismatch, no coverage driver, `--random-order-seed` without random order - make the run exit 1 even when every test passed. - `failOnAllIssues="true"` turns every switch on; `--do-not-fail-on-risky` and its siblings switch one off again from the command line. The defaults are deliberate: a fresh install on a legacy codebase should report problems without breaking on its first run. They are also why a CI pipeline can stay green for months while the output fills with risky tests and warnings nobody reads. ## `failOnWarning` versus `failOnPhpunitWarning` These are easy to confuse: 1. `failOnWarning` is about **PHP warnings triggered while tests run** - `E_WARNING` from `file_get_contents()` on a missing file, `E_USER_WARNING` from `trigger_error()`. Default false. 2. `failOnPhpunitWarning` is about **PHPUnit's own warnings** about configuration and metadata. Default true. `<source>` narrows the first kind: with `restrictWarnings="true"`, only warnings from first-party code are reported, so a noisy dependency does not dominate the output. ## Reading the summary line The last lines of the output tell you which rule applied: - `OK (40 tests, 95 assertions)` - nothing to report, exit code 0. - `OK, but there were issues!` - tests passed, but risky tests, warnings, notices or deprecations were recorded; the exit code depends on the `failOn*` switches. - `OK, but some tests were skipped!` - skipped or incomplete tests without other issues. - `FAILURES!` or `ERRORS!` - at least one failure or error; exit code 1 or 2. Because the first three can all end with exit code 0, a CI job that only checks the exit code cannot tell a clean run from one full of risky tests. That is the practical argument for turning the relevant switches on rather than relying on someone to read the log. ## A CI setup for a legacy project 1. Run once with the defaults and read the issue list. 2. Fix or quarantine assertion-less tests; add `#[DoesNotPerformAssertions]` only where "does not throw" really is the test. 3. Turn on `failOnRisky="true"` and `failOnWarning="true"`, or `failOnAllIssues="true"`. 4. Add `beStrictAboutOutputDuringTests="true"` so stray `echo` and `var_dump()` calls surface. 5. Use `--display-all-issues` (or the matching `displayDetailsOn*` attributes) so CI logs show details, not just counts. The generated configuration from `--generate-configuration` already sets `failOnRisky` and `failOnWarning` to true, a hint of what the maintainers consider a sensible baseline.

  • With PHPUnit 13, a CI job fails with exit code 1 although every test passed. What is a likely cause?
    A PHPUnit warning: `failOnPhpunitWarning` defaults to true, so warnings about the run itself fail it. Typical ones are a configuration that does not validate, a data set with more arguments than the method accepts, a missing coverage driver when a report was requested, or `--random-order-seed` without random order. The warning list at the end of the output names it.
  • In PHPUnit 13, why does beStrictAboutChangesToGlobalState seem to do nothing on its own?
    PHPUnit can only detect a change if it took a snapshot before the test. It snapshots globals and static properties only when `backupGlobals` or `backupStaticProperties` is enabled. With neither on, there is nothing to compare, so the check never reports anything.

saying these in an interview costs you the question

  • Believes a risky test fails the build by default
  • Thinks failOnWarning covers PHPUnit's own configuration warnings
  • Expects exit code 0 whenever every test passed
  • Silences assertion-less tests with #[DoesNotPerformAssertions] everywhere
  • Assumes the global-state check works without enabling backups