skip to content

What does Kotest's assertionMode setting do when set to AssertionMode.Error in project config, and which test bugs does it catch?

level: seniorimportance: should knowfreq 24%

answer

  1. counts Kotest assertions per test
  2. None / Warn / Error
  3. zero assertions + Error = test fails
  4. blind to other libraries and hand-rolled checks
  5. roll out Warn → triage → Error

basics

~20 s

Kotest counts assertions made through its own assertion library during a test. With AssertionMode.Error, a test that completes without executing any counted assertion fails; Warn logs instead; None disables the check. It catches tests that verify nothing.

solid answer

~50 s

Kotest's assertion library increments a counter each time an assertion executes. `assertionMode` in `AbstractProjectConfig` decides what happens when a test finishes with that counter at zero: - `AssertionMode.None` — nothing (the permissive setting). - `AssertionMode.Warn` — a warning is emitted. - `AssertionMode.Error` — the test fails. What it catches is the test that *looks* fine and proves nothing: an assertion left off after a refactor, a test that calls the system under test and forgets to check the result, a test whose only assertions live inside a lambda that is never invoked, a stubbed-out placeholder someone meant to finish. The limits matter as much as the feature. Only Kotest's own assertions are counted — a hand-rolled `if (x != y) throw`, or assertions from another library, are invisible to the counter, so those tests would be flagged despite verifying something real. That is why teams usually roll it out as `Warn` first, fix or exempt the noisy specs, then move to `Error`.

code

kotlin · 11 lines
kotlin
package io.kotest.provided

import io.kotest.core.config.AbstractProjectConfig
import io.kotest.assertions.AssertionMode

class ProjectConfig : AbstractProjectConfig() {
    override val assertionMode = AssertionMode.Error
}

// flagged under Error: no Kotest assertion executes
// test("rejects an expired token") { validator.validate(expiredToken) }

go deeper

for a junior

Know the three modes and that Error fails a test that made no Kotest assertion.

for a middle

Explain the counting mechanism and give examples of the bugs it catches, including empty-loop assertions.

for a senior

Add the false-positive class (non-Kotest assertions) and describe a staged rollout from Warn to Error.

for a principal

Argue why this is a model global default — uniform, loud, invariant-encoding — and contrast it with globals that change semantics silently.

## The problem it solves The worst test is not the failing one; it is the passing one that asserts nothing. It looks like coverage, it looks like verification, and it will never go red no matter how badly the code breaks. These appear naturally: ```kotlin test("rejects an expired token") { validator.validate(expiredToken) // and… nothing } ``` Someone deleted the assertion while debugging, or wrote the arrangement first and never came back, or the assertion sits inside a `forEach` over an empty list. All pass. All are worthless. ## The mechanism Kotest's assertion library keeps a count of assertions executed within a test. Every matcher call through that library (a `shouldBe`, a `shouldThrow`, a `should` with a matcher) bumps it. When the test completes, Kotest inspects the count and applies the configured `assertionMode`: ```kotlin package io.kotest.provided class ProjectConfig : AbstractProjectConfig() { override val assertionMode = AssertionMode.Error } ``` - **`None`** — the check is off. Any test may assert nothing. - **`Warn`** — assertion-free tests emit a warning but still pass. This is the migration setting. - **`Error`** — assertion-free tests fail. The setting is a project-level default, and like other defaults it can be relaxed for a spec that legitimately needs it. ## What it catches - Tests where the assertion was removed and not restored. - Placeholder tests that only exercise the system under test. - Tests whose assertions are inside a loop or lambda that never executes — the empty-collection case, which is the sneakiest of the family because the code *looks* like it asserts. - Tests where an early `return` or a swallowed exception skips the assertion. That third category is the one that justifies the setting on its own: a property or table-driven style test iterating an accidentally empty data set is invisible to review and to coverage tools, but not to an assertion counter. ## What it does not catch - **Weak assertions.** `result shouldNotBe null` counts. The check is about *presence*, not quality. - **Assertions from other libraries or hand-rolled checks.** The counter only knows Kotest's own assertion library, so a test doing `if (actual != expected) error("...")` counts as assertion-free and will be flagged even though it genuinely verifies something. - **Tests that assert the wrong thing.** It cannot read intent. ## Rolling it out On an existing suite, switching straight to `Error` produces a wall of failures mixing genuine no-op tests with tests using other assertion styles. The workable sequence: 1. Turn on `Warn` project-wide and collect the list. 2. Triage: delete or fix genuine no-op tests (usually a satisfying cull), and convert other-library assertions to Kotest matchers where cheap. 3. Exempt the specs that legitimately cannot use Kotest assertions, relaxing the mode for those specs only. 4. Flip the project default to `Error` so new no-op tests cannot land. Step 4 is the point of the exercise. `Warn` alone decays — warnings scroll past in CI logs and nobody reads them. The value comes from the setting being enforcing, which is exactly why it belongs in project config rather than being left to each spec. ## Why project config is the right home This is the archetype of a good global default: it encodes a suite-wide quality invariant, it should be uniform (a no-op test is no more acceptable in one module than another), and it fails loudly rather than changing behaviour subtly. Compare that with a global isolation mode, which changes semantics invisibly — the same mechanism, very different risk profile. ## Interview signal Strong answers describe the counting mechanism rather than treating it as magic, name the three modes, and volunteer both a limitation (only Kotest assertions are counted; weak assertions still pass) and a migration path (`Warn` → triage → `Error`).

  • Would AssertionMode.Error flag a test that verifies behaviour with a hand-written `if (actual != expected) error(...)` check?
    Yes, and that is its main false positive. The counter only sees assertions made through Kotest's assertion library, so a hand-rolled check or another library's assertion leaves the count at zero and the test is reported as assertion-free. The remedies are converting those checks to Kotest matchers or relaxing the mode for the specs that genuinely cannot.
  • How would you introduce this setting on a suite with 4,000 existing tests?
    Start at Warn project-wide and collect the flagged list rather than fixing blind. Triage it: genuine no-op tests get deleted or completed, other-library assertions get converted where cheap, and the remainder get an explicit per-spec relaxation with a comment. Then flip the project default to Error so no new assertion-free test can be merged — the enforcement step is what makes the effort stick.

saying these in an interview costs you the question

  • Believing it evaluates assertion quality rather than presence
  • Assuming it counts assertions from any assertion library
  • Turning it straight to Error on a legacy suite and drowning in false positives
  • Thinking a warning-only mode is sufficient enforcement over time
  • Confusing it with soft assertion collection, which is about grouping failures rather than requiring one

context