A unittest suite reports 40 skips on CI and none locally. How do you find what silently stopped running?

seniorshould knowfreq 28%

answer

Green does not mean everything ran
The output already tells you why
Group the reasons before chasing one
One decorator can remove a whole class
Baseline the count so growth turns red

basics

~20 s

Run the suite verbosely so every skip prints next to its reason, then trace each reason back to the condition that produced it. Skips are almost always import-time environment checks, so compare the CI image's platform, environment variables and optional capabilities against the local machine.

solid answer

~40 s

Start with the reasons: python -m unittest -v prints each skipped test with the string its decorator supplied, which is the entire reason those strings must name something specific. Group them — a block of 30 from one file almost always means a class-level @unittest.skipUnless whose probe went false, since one decorator on a unittest.TestCase subclass disables every method inside it. Then check what the probe reads: sys.platform, an os.environ flag, or a capability check that quietly returns false in a slimmer container image. Confirm by asserting the environment instead of inferring it — in the job where a capability is required, fail loudly rather than skip. Finally, hold the line: record a baseline skip count and fail the build when it grows, because skips never turn a run red on their own.

Start with the reasons: `python -m unittest -v` prints each skipped test with the string its decorator supplied, which is the entire reason those strings must name something specific. Group them — a block of 30 from one file almost always means a class-level `@unittest.skipUnless` whose probe went false, since one decorator on a `unittest.TestCase` subclass disables every method inside it. Then check what the probe reads: `sys.platform`, an `os.environ` flag, or a capability check that quietly returns false in a slimmer container image. Confirm by asserting the environment instead of inferring it — in the job where a capability is *required*, fail loudly rather than skip. Finally, hold the line: record a baseline skip count and fail the build when it grows, because skips never turn a run red on their own.

code

python · 17 lines

python

import importlib.util
import unittest

HAS_CODEC = importlib.util.find_spec("lzma") is not None


@unittest.skipUnless(HAS_CODEC, "optional codec not present in this interpreter build")
class CompressedBatchTests(unittest.TestCase):
    def test_batch_ordering(self):
        self.assertTrue(HAS_CODEC)

    def test_batch_round_trip(self):
        self.assertTrue(HAS_CODEC)


print("HAS_CODEC:", HAS_CODEC)
unittest.main(argv=["prog"], verbosity=2, exit=False)

go deeper

for a junior

Know that a skipped test does not run and does not fail the build, and that running the suite verbosely prints each skip with the reason someone wrote. That output is the first place to look.

for a middle

Explain that skipIf/skipUnless conditions are evaluated at import, so you can reproduce them by importing the module in the failing environment and printing the probe. Note that a class-level decorator skips every method inside.

for a senior

Demonstrate the operating judgement: separate legitimate platform exclusions from accidental gaps, make a required capability an assertion rather than a skip in the job that needs it, and baseline the skip count so drift becomes visible.

for a principal

Own the policy — who may add a skip and on what terms, what the suite's skip and expected-failure counts mean as a release signal, and how the CI image is versioned so a base-image change cannot silently delete coverage.

Why this is a real production problem

Skips do not fail a build. unittest.TestResult.wasSuccessful() returns True with any number of them, so a suite can lose most of its coverage while every commit reports green. The typical shape: an ordering test for a 6,800-row batch in a document-conversion queue sat behind a capability probe, the CI base image was slimmed down six months ago, the probe went false, and a regression in batch ordering shipped past a "passing" suite. Nobody lied; nobody looked at the skip count.

Step 1 — read the reasons

The first move is always the verbose run:

$ python -m unittest discover -v
test_pages_keep_input_order (tests.test_batches.ConverterTests) ... skipped 'optional codec not present'

Each skip prints its reason string. This is where the discipline of writing a specific reason pays for itself: "optional codec not present" points straight at the container image, while "flaky" or an empty reason tells you nothing and costs you an afternoon.

Step 2 — group them, and look for class-level skips

Forty skips are rarely forty decisions. Applied to a unittest.TestCase subclass rather than a method, one @unittest.skipUnless disables every test in it. So group the skipped test ids by module and class: one reason repeated thirty times is one condition to investigate, and a class or module-level skip is the highest-leverage thing to find because it removes the most coverage per line of source.

Raising unittest.SkipTest at module import level is the extreme version — it takes out a whole file, and the file's tests never even appear individually.

Step 3 — reproduce the condition, not the test

Because skipIf/skipUnless conditions are evaluated at import time, they are trivially reproducible without running anything. Import the module in the CI environment and print the probe:

$ python -c "import tests.test_batches as m; print(m.HAS_CODEC)"
False

Compare against the same command locally. The differences that produce this are a short list:

platform — a sys.platform guard that is correct on one OS and over-broad on another;
environment variables — an os.environ feature flag that the CI job never sets, or sets to a string like "false" that is nonetheless truthy;
optional capabilities — a module or codec present in a developer install and absent from a slim image;
an import-time probe that swallows its own error, turning a genuine misconfiguration into a quiet False.

That last one is the nastiest, and it is a design defect in the test, not in the environment.

Step 4 — decide, per skip, which kind it is

Every skip is exactly one of two things, and they need opposite treatment:

A legitimate exclusion. The test genuinely does not apply here — a POSIX-only path on Windows, a feature the interpreter build does not provide. This should stay forever, and its reason should say so.
An accidental gap. The test should run in this job and does not. The fix is not to make the skip prettier; it is to make the capability present and then make its absence an error. In a job whose entire purpose is exercising that path, a missing capability is a broken job, so the probe should raise rather than skip.

Writing that distinction into the reason string — "excluded on Windows" versus "requires the X capability" — lets the next person triage from the output alone.

Step 5 — stop it recurring

Because the runner will never go red over this, add the pressure yourself:

Baseline the count. Record the expected number of skips per job and fail when it rises. A new skip then needs a deliberate decision rather than a silent drift.
Assert the environment in a smoke test. One test per job that asserts the capabilities that job is supposed to have, with a hard assertion rather than a skip. If the image regresses, that test fails immediately, instead of forty others quietly disappearing.
Review class-level skips harder than method-level ones. They are one line and remove the most.
Publish the reasons. The verbose output's reason column, collected per job, is the artefact that makes this visible; a summary nobody reads is how six months went by in the first place.

The underlying lesson is that a skip is a silent signal by design, so it needs an explicit consumer. A suite that never inspects its own skip list is trusting a number it does not read.

### Why this is a real production problem Skips do not fail a build. `unittest.TestResult.wasSuccessful()` returns `True` with any number of them, so a suite can lose most of its coverage while every commit reports green. The typical shape: an ordering test for a 6,800-row batch in a document-conversion queue sat behind a capability probe, the CI base image was slimmed down six months ago, the probe went false, and a regression in batch ordering shipped past a "passing" suite. Nobody lied; nobody looked at the skip count. ### Step 1 — read the reasons The first move is always the verbose run: ``` $ python -m unittest discover -v test_pages_keep_input_order (tests.test_batches.ConverterTests) ... skipped 'optional codec not present' ``` Each skip prints its reason string. This is where the discipline of writing a *specific* reason pays for itself: `"optional codec not present"` points straight at the container image, while `"flaky"` or an empty reason tells you nothing and costs you an afternoon. ### Step 2 — group them, and look for class-level skips Forty skips are rarely forty decisions. Applied to a `unittest.TestCase` subclass rather than a method, one `@unittest.skipUnless` disables every test in it. So group the skipped test ids by module and class: one reason repeated thirty times is one condition to investigate, and a class or module-level skip is the highest-leverage thing to find because it removes the most coverage per line of source. Raising `unittest.SkipTest` at module import level is the extreme version — it takes out a whole file, and the file's tests never even appear individually. ### Step 3 — reproduce the condition, not the test Because `skipIf`/`skipUnless` conditions are evaluated at import time, they are trivially reproducible without running anything. Import the module in the CI environment and print the probe: ``` $ python -c "import tests.test_batches as m; print(m.HAS_CODEC)" False ``` Compare against the same command locally. The differences that produce this are a short list: * **platform** — a `sys.platform` guard that is correct on one OS and over-broad on another; * **environment variables** — an `os.environ` feature flag that the CI job never sets, or sets to a string like `"false"` that is nonetheless truthy; * **optional capabilities** — a module or codec present in a developer install and absent from a slim image; * **an import-time probe that swallows its own error**, turning a genuine misconfiguration into a quiet `False`. That last one is the nastiest, and it is a design defect in the test, not in the environment. ### Step 4 — decide, per skip, which kind it is Every skip is exactly one of two things, and they need opposite treatment: * **A legitimate exclusion.** The test genuinely does not apply here — a POSIX-only path on Windows, a feature the interpreter build does not provide. This should stay forever, and its reason should say so. * **An accidental gap.** The test *should* run in this job and does not. The fix is not to make the skip prettier; it is to make the capability present and then make its absence an error. In a job whose entire purpose is exercising that path, a missing capability is a broken job, so the probe should raise rather than skip. Writing that distinction into the reason string — "excluded on Windows" versus "requires the X capability" — lets the next person triage from the output alone. ### Step 5 — stop it recurring Because the runner will never go red over this, add the pressure yourself: * **Baseline the count.** Record the expected number of skips per job and fail when it rises. A new skip then needs a deliberate decision rather than a silent drift. * **Assert the environment in a smoke test.** One test per job that asserts the capabilities that job is supposed to have, with a hard assertion rather than a skip. If the image regresses, that test fails immediately, instead of forty others quietly disappearing. * **Review class-level skips harder than method-level ones.** They are one line and remove the most. * **Publish the reasons.** The verbose output's reason column, collected per job, is the artefact that makes this visible; a summary nobody reads is how six months went by in the first place. The underlying lesson is that a skip is a *silent* signal by design, so it needs an explicit consumer. A suite that never inspects its own skip list is trusting a number it does not read.

How would you make a growing skip count actually break a build?
Run the suite with a result object you inspect yourself — the collected skips are available on the result the runner returns — and compare the count, or the set of skipped test ids, against a checked-in baseline. Exceeding it exits non-zero. Pinning the *ids* rather than the number is stronger, because it catches a swap where one test starts skipping as another stops.
A skip reason just says 'flaky'. What do you do with it?
Treat it as unowned debt. Find the commit that added it, attach a real cause or a ticket, and give it an owner and a date. If nobody can say what it protects or why it was unreliable, the honest options are to fix the nondeterminism or delete the test — leaving it skipped forever is the one choice that keeps the cost without the benefit.
Why does an environment-dependent skip belong in a decorator rather than in the test body?
Because it is a property of the process, evaluated once at import, and stating it declaratively next to the test name makes the exclusion visible to a reader and to a grep over the source. Run-time skipping inside a body hides the condition where only an execution reveals it, and it costs the setup work the test already performed.

saying these in an interview costs you the question

Assumes a green run means every test executed
Chases individual skips without grouping the reasons
Overlooks that one class-level decorator skips many tests
Suggests deleting the skipped tests to clean up the output
Never checks whether the CI image differs from the laptop
Relies on the runner to turn red as skips accumulate

A unittest suite reports 40 skips on CI and none locally. How do you find what silently stopped running?

§ answer

basics

solid answer

code

§ go deeper

▾level notes4 levels

▾deep dive~4 min

Why this is a real production problem

Step 1 — read the reasons

Step 2 — group them, and look for class-level skips

Step 3 — reproduce the condition, not the test

Step 4 — decide, per skip, which kind it is

Step 5 — stop it recurring

▾common follow-ups3 questions

▾red flags6 flags

§ context

▾related questions5 questions

▾related topics4 topics

Why this is a real production problem

Step 1 — read the reasons

Step 2 — group them, and look for class-level skips

Step 3 — reproduce the condition, not the test

Step 4 — decide, per skip, which kind it is

Step 5 — stop it recurring

answer

go deeper

context