skip to content

How do you build and run a unittest suite programmatically with TestLoader and TextTestRunner?

level: middleimportance: nice to knowfreq 20%

answer

  1. Three objects, one job each
  2. One finds, one holds, one runs
  3. The runner hands back a result
  4. loader, suite, runner, wasSuccessful

basics

~10 s

Ask a unittest.TestLoader for tests, collect them in a unittest.TestSuite, and hand that to unittest.TextTestRunner().run(suite). The returned result object's wasSuccessful() gives you the exit status.

solid answer

~50 s

The three objects behind the command line are separable. `unittest.TestLoader` turns inputs into tests — `loadTestsFromTestCase` for one class, `loadTestsFromModule` for a module, `loadTestsFromName` for a dotted string, `discover` for a tree. `unittest.TestSuite` is a composite you fill with `addTest` and `addTests`, and suites nest. `unittest.TextTestRunner` executes a suite and writes the familiar dots and summary to stderr; its `run` returns a `unittest.TestResult` whose `wasSuccessful()` you turn into a process exit code. That is the whole harness in five lines. Reach for it when you need something the CLI cannot express — assembling a subset from data, or running a suite from inside another program. When the customisation belongs to a test module rather than to a script, the `load_tests` hook is the better lever: define `load_tests(loader, standard_tests, pattern)` in the module and both discovery and `unittest.main()` will call it.

code

python · 19 lines
python
import sys
import unittest


class CsvImportTests(unittest.TestCase):
    def test_header_is_validated(self):
        self.assertIn("employee_id", "employee_id,hours,rate")

    def test_rejects_negative_hours(self):
        with self.assertRaises(ValueError):
            int("-")


loader = unittest.TestLoader()
suite = unittest.TestSuite()
suite.addTests(loader.loadTestsFromTestCase(CsvImportTests))

result = unittest.TextTestRunner(verbosity=2).run(suite)
sys.exit(0 if result.wasSuccessful() else 1)

go deeper

for a junior

Know that the command line is not magic: a loader finds tests, a suite holds them, a runner executes them. Recognising those three names in a script is enough at this level.

for a middle

Be able to write the five-line harness and explain each object's single job, including that the runner returns a result rather than exiting, and that unittest.main() calls sys.exit on your behalf.

for a senior

Show judgement about when a custom driver is warranted at all, and prefer the load_tests hook when the customisation belongs to the test module. Explain the risk of a second entry point drifting from what CI executes.

for a principal

Own the entry-point question. A bespoke harness is code to maintain and a way for local and CI runs to diverge; argue for one command, with data-driven suites expressed through load_tests rather than through a parallel runner nobody else knows about.

`python -m unittest` is convenience over an ordinary object API, and knowing that API is what lets you build a harness when the command line runs out. ### The three roles **`unittest.TestLoader` produces tests.** Its methods differ only in what they accept: `loadTestsFromTestCase(cls)` takes a `unittest.TestCase` subclass and returns a suite of its `test`-prefixed methods; `loadTestsFromModule(mod)` takes an imported module; `loadTestsFromName(name, module=None)` takes the same dotted string the command line accepts; `discover(start_dir, pattern, top_level_dir)` runs the directory walk. Two class attributes tune it: `testMethodPrefix`, which defaults to `'test'`, and `sortTestMethodsUsing`, which decides the order of methods within a class. `unittest.defaultTestLoader` is a ready-made shared instance. **`unittest.TestSuite` holds them.** It is a composite: it contains tests *and* other suites, and iterating one yields the leaves. Build it with `addTest` for one item or `addTests` for an iterable. Nothing about a suite runs anything — it is a plan. **`unittest.TextTestRunner` executes.** Construct it with options such as `verbosity=2`, `failfast=True` and `buffer=True` — the direct equivalents of `-v`, `-f` and `-b` — then call `run(suite)`. It returns a `unittest.TestResult` carrying counts and the collected failures and errors, and exposing `wasSuccessful()`. The runner writes its report to stderr and never exits the process, which is the important difference from the next section. ### Versus `unittest.main()` `unittest.main()` is a full command-line program in a function: it parses `sys.argv`, applies `-v`, `-k` and friends, loads tests from the calling module by default, runs them, and then calls `sys.exit` with a status derived from the result. That last step is why embedding `unittest.main()` inside a larger program is a trap — it will terminate the process. Driving the loader, suite and runner yourself gives you the result object and lets you decide what to do with it, including exiting with your own code. ### When to use each Prefer the command line. Reach for the object API only when it buys something real: - Building a suite from data — for example, a payroll import checked against a table of CSV fixtures, where the cases are rows rather than hand-written methods. - Running tests from inside another program, such as a self-check subcommand of a tool. - Wrapping the run with setup or reporting that has to happen in the same process. When the customisation logically belongs to a *test module* rather than to a driver script, use the `load_tests` protocol instead. Define `load_tests(loader, standard_tests, pattern)` at module level, return the suite you want, and both discovery and `unittest.main()` will call it in place of the default scan. Placed in a package `__init__.py`, the same hook customises discovery for that whole package. It keeps the customisation next to the tests and preserves one entry point, which a bespoke runner script does not. ### Pitfalls The suite is a plan, so a mistake there is silent: adding a `unittest.TestCase` *class* instead of the suite returned by `loadTestsFromTestCase` gives you something that does not run as you expect. Forgetting to check `wasSuccessful()` gives a script that exits zero on a red suite, which is worse than no script at all. And a hand-rolled runner drifts from what CI runs unless it is the same code path — if you build one, make it the only entry point rather than a second one. ### What to say in an interview Name the three objects and their one job each, note that `run` returns a result rather than exiting, and say when you would use `load_tests` instead of a script. The senior signal is arguing *against* a custom runner in most cases: the standard entry point keeps local runs and CI identical, and every bespoke harness is a thing to maintain.

  • How does `unittest.main()` differ from driving TextTestRunner yourself?
    `unittest.main()` parses `sys.argv`, loads tests from the calling module, runs them and then calls `sys.exit` with a status derived from the outcome. Driving the runner yourself skips the argument parsing and, crucially, hands back a `unittest.TestResult` instead of terminating the process — so you can inspect counts, report elsewhere, or choose your own exit code. Embedding `unittest.main()` in a larger program will kill it.
  • When would you use the `load_tests` protocol instead of a runner script?
    When the customisation belongs to the tests rather than to a driver — building cases from a data table, or filtering what a package exposes. Defining `load_tests(loader, standard_tests, pattern)` in a module, or in a package `__init__.py` for the whole package, means both discovery and `unittest.main()` honour it. The customisation stays next to the tests and there is still exactly one entry point, which a bespoke script gives up.
  • What does `unittest.TextTestRunner.run()` give you back?
    A `unittest.TestResult` describing the run: the number of tests executed, and the collected failures, errors, skips and expected failures, with `wasSuccessful()` as the single boolean verdict. It is the object you translate into an exit status or a report. The runner itself only writes its human-readable summary to stderr and leaves the process alive.

saying these in an interview costs you the question

  • Adding a TestCase class to a suite instead of the loaded tests
  • Expecting TextTestRunner.run to exit the process by itself
  • Ignoring wasSuccessful so a red suite still exits zero
  • Calling unittest.main inside a larger program that must keep running
  • Reaching for a bespoke runner where the command line already suffices
  • Assuming a suite runs its tests as soon as they are added

context