skip to content

Python Test Design

The judgment calls behind a Python suite — what to fake versus inject, why patching global state is a smell, how to keep tests order-independent. Senior candidates get the open-ended version of these.

part ofPythonoverview, primer and where to startread it →
on this pageshow

questions

21

Why does a test that calls an async def function without awaiting it still pass?

level: juniorimportance: must knowfreq 55%

answer

  1. Nothing actually ran
  2. The call returns an object
  3. Truthy object, passing assertion
  4. RuntimeWarning fires at collection time
  5. Async runner plus warnings as errors

basics

~20 s

Calling an async def function only builds a coroutine object; the body never runs. Assertions then inspect that object rather than a result, so the test passes vacuously. CPython only warns, at garbage-collection time, that the coroutine was never awaited.

solid answer

~40 s

`async def` defines a coroutine *function*: calling it constructs a coroutine object and returns immediately without executing a line of the body. A coroutine object is truthy and not `None`, so `assertIsNotNone` or `assert result` passes while the code under test never ran. The only warning is `RuntimeWarning: coroutine ... was never awaited`, raised when the object is garbage-collected — it does not fail the build and may be attributed to a later test. Fixes: run async test bodies under an async-aware runner such as `unittest.IsolatedAsyncioTestCase` (or `asyncio.run`), prove once that a deliberately failing async test actually goes red, run the suite with `python -X dev` or warnings as errors, and assert on concrete values rather than on objects being non-null.

code

python · 14 lines
python
import warnings


async def fetch_count() -> int:
    return 42


with warnings.catch_warnings(record=True) as caught:
    warnings.simplefilter("always")
    result = fetch_count()      # no await, so the body never runs
    assert result is not None   # the "assertion" only sees a coroutine object
    del result

print([type(w.message).__name__ for w in caught])

go deeper

for a junior

Be ready to say what calling an async def function returns before any await happens, and why an assertion on that object can pass while nothing ran. Knowing the phrase 'coroutine was never awaited' is expected.

for a middle

Explain the mechanics: coroutine object construction, the runner that must drive it, and why the RuntimeWarning appears at garbage-collection time rather than at the call — so it can be blamed on the wrong test.

for a senior

Show the suite-level judgement: prove the runner really awaits async tests, promote the warning to an error in CI, and insist on value assertions so a missing await cannot produce a green run.

for a principal

Own the policy angle: a suite where a failing async test can pass is a false safety signal across every team using it. Decide where warnings-as-errors is enforced and how new async test infrastructure is verified.

### Calling and running are two different events `async def` does not define a function that executes when you call it. It defines a **coroutine function**, and calling it constructs a **coroutine object** and returns immediately. Not one line of the body has run at that point. The body runs only when something *drives* the coroutine: an `await` from inside another coroutine, `asyncio.run`, a task created with `asyncio.create_task`, or an async-aware test runner such as `unittest.IsolatedAsyncioTestCase`. That distinction is what produces the most dangerous failure mode in an async test suite: a test that is green about nothing. ### How the vacuous pass happens A synchronous test method calls the async code under test and asserts on the result: ```python def test_pick_list_is_built(self): result = build_pick_list("AISLE-04") # returns a coroutine object self.assertIsNotNone(result) # always true ``` `result` is a coroutine object. It is truthy, it is not `None`, it has a `__name__`, and it happily survives `assertIsNotNone`, `assertTrue`, and any `assert x` you write. The builder never ran, no warehouse bin was ever read, no exception could possibly be raised — and the test reports success. The same shape appears in a subtler form when the test method itself is `async def` but the runner is not async-aware: the runner calls the method, gets a coroutine object back, treats a non-raising call as a pass, and discards it. ### The only signal CPython gives you When the unawaited coroutine object is finally destroyed, CPython emits `RuntimeWarning: coroutine 'build_pick_list' was never awaited`. Three properties of that warning matter for test design: 1. **It is a warning, not an error.** By default it is printed once per location and the process exit status is unaffected, so a CI job stays green. 2. **It fires at destruction time, not at call time.** Under reference counting that is usually soon after the last reference drops, but if the coroutine object is captured in a local that outlives the test, or in a cycle awaiting the cyclic collector, the warning can be attributed to a *later* test — or to interpreter shutdown, where nobody reads it. 3. **It says nothing about the assertion that lied.** The test that passed and the warning that was printed are two separate lines of output, often far apart. ### Designing the failure out Three habits remove this class of bug: **Run every async test body under a real runner.** The test's asynchronous body must be awaited by something that owns an event loop for the duration of the test. If a test is `async def`, verify that the runner you use actually awaits it — write a deliberately failing async test once and confirm it goes red. A test suite where a failing async test passes is worth exactly nothing, and that check takes thirty seconds. **Turn the warning into an error.** Running the suite with `python -X dev`, or with warnings configured as errors for `RuntimeWarning`, converts silent leakage into a failure. Because the warning is raised at collection time, pair it with a per-test cleanup that drops references and, where you need determinism, forces a collection so the warning is attributed to the test that caused it rather than to its successor. **Assert on values, not on objects.** `assertIsNotNone(result)` passes for a coroutine object; `self.assertEqual(result, ["AISLE-04/BIN-12"])` cannot. Assertions that pin an actual value are immune to this failure mode, because a coroutine object never equals the data you expected. This is also why a helper that *returns* a coroutine rather than awaiting it is a hazard at every call site: each caller must remember the `await`, and forgetting it fails silently rather than loudly. ### Why interviewers ask it The question separates candidates who have internalised that `async def` is a *factory for suspendable work* from those who read `async` as a decoration that makes a function faster. It also probes test-suite judgement: a candidate who says "the test would fail with a TypeError" has never watched an async suite go green while doing nothing, and a candidate who shrugs at the never-awaited warning has not yet been burned by it. The strong answer names the coroutine object, names the RuntimeWarning and its garbage-collection timing, and closes with the design fix — an async-aware runner, warnings as errors, and assertions on values.

  • How do you make the never-awaited RuntimeWarning fail the suite instead of scrolling past?
    Run the suite with `python -X dev`, or configure warnings so `RuntimeWarning` is raised as an error. Because the warning fires when the coroutine object is destroyed, also drop references in per-test cleanup — otherwise the failure lands on whichever test happens to be running when the collector gets to it, and you debug the wrong test.
  • Why does asserting the result is not None hide this bug, and what assertion would not?
    A coroutine object is a perfectly ordinary object: truthy, not `None`, with a `__name__`. Any assertion that only checks existence passes. An assertion that pins an actual value — comparing against the expected list, string or number — cannot pass, because a coroutine object never equals the data you expected.
  • How would you prove your async tests are actually being run at all?
    Write one async test that asserts something plainly false and confirm the suite goes red. If it passes, the runner is calling your test method and discarding the coroutine it returns, and every async test in the suite is currently worthless.

Calling an async def function is like filling in an order form: you now hold the form, but nothing has been picked. Awaiting it is handing the form to someone who does the work.

saying these in an interview costs you the question

  • Says the body starts running as soon as you call the function
  • Believes a green async test proves the code under test ran
  • Treats the never-awaited RuntimeWarning as harmless noise
  • Expects a TypeError when a coroutine is not awaited
  • Assumes any test runner will await a returned coroutine
  • Thinks assertIsNotNone meaningfully checks an async result

context

open as a page

Why is a collaborator constructed inside a Python class harder to test than one passed in?

level: juniorimportance: must knowfreq 60%

basics

~20 s

A collaborator built inside the class gives a test no way in, so the only override left is patching the name the module imported. Take the collaborator as a constructor parameter and the test simply passes a small fake object.

open as a page

Why does assigning over a module attribute in a test leak into later tests?

level: juniorimportance: must knowfreq 62%

basics

~20 s

An imported module exists once per process, cached in sys.modules, so rebinding one of its attributes changes it for every user until something restores it. A plain assignment has no rollback, and a failing assertion skips the restore line.

open as a page

Why does asyncio.sleep as a synchronisation device make an async test flaky?

level: middleimportance: must knowfreq 58%

basics

~10 s

A sleep guesses how long another coroutine needs, and a loaded machine breaks the guess. Await an asyncio.Event the code under test sets instead: the handshake resumes as soon as the state is real.

open as a page

How do tempfile.TemporaryDirectory and unittest's addCleanup keep a test's scratch files isolated?

level: middleimportance: must knowfreq 60%

basics

~10 s

tempfile.TemporaryDirectory creates a fresh, uniquely named directory per test, so no two tests share a path. Registering its cleanup with TestCase.addCleanup deletes the tree even when setUp or the test itself raises.

open as a page

Why does calling time.monotonic() inline make a function hard to test deterministically?

level: middleimportance: must knowfreq 55%

basics

~20 s

Because the function reaches into a process-wide clock the test cannot control, so its timing branches only fire after real time passes. Accept a clock callable parameter defaulting to time.monotonic and pass a fake in the test.

open as a page

Fifteen unittest.mock.patch decorators stack on one test for an invoice-PDF renderer. What does that say about the design?

level: seniorimportance: must knowfreq 50%

basics

~20 s

It says the renderer builds or imports all fifteen collaborators itself, leaving the test no way in but the module namespace. Pass them in instead, swap the patches for two or three fakes, and split the class if the count stays high.

open as a page

How does contextlib.redirect_stdout let a test capture what a function prints?

level: juniorimportance: should knowfreq 55%

basics

~10 s

contextlib.redirect_stdout rebinds sys.stdout to any writable text object for the duration of its with block. Point it at an io.StringIO, run the code, then assert on that buffer's getvalue() text.

open as a page

How do you make code that draws from Python's random module reproducible in a test?

level: juniorimportance: should knowfreq 50%

basics

~10 s

Give the code its own generator: build random.Random(seed) and pass it in, then draw with rng.random(), rng.choice() and friends. Calling random.seed() instead mutates one process-wide generator that every other caller shares.

open as a page

What goes wrong with `def render(doc, store=PdfStore())` as an injection seam in Python?

level: middleimportance: should knowfreq 45%

basics

~20 s

The default expression runs once, when the def statement executes at import time, so every call that omits the argument shares one collaborator built before any test ran. Default to None and construct inside the body instead.

open as a page

How does unittest.mock.patch.dict isolate os.environ changes to one test?

level: middleimportance: should knowfreq 46%

basics

~20 s

It copies the mapping's contents on entry, applies your overrides in place, and on exit clears the mapping and writes the copy back. Anything the block added, changed or removed is undone, even when the test raises.

open as a page

Why can't unittest.mock.patch replace datetime.datetime.now directly?

level: middleimportance: should knowfreq 45%

basics

~10 s

datetime.datetime is a C-implemented type whose attributes cannot be assigned, so patch's setattr raises TypeError. Inject a now() callable instead, or rebind the module-level name to a pure-Python subclass that overrides now.

open as a page

How do you test that a coroutine's cancellation and timeout branches actually run?

level: seniorimportance: should knowfreq 42%

basics

~10 s

Start the coroutine as a task, await an event so it is running, then cancel and await it, asserting task.cancelled() and the cleanup. Test the timeout branch with a tiny asyncio.timeout budget.

open as a page

What goes wrong when an async test leaves asyncio tasks pending at teardown?

level: seniorimportance: should knowfreq 36%

basics

~10 s

The test finishes but its work does not: CPython logs 'Task was destroyed but it is pending', any exception the task raised is lost, and it keeps memory alive. Cancel and await every task.

open as a page

A translation-memory updater bakes its output path in at import time. What seam makes its rollback testable?

level: seniorimportance: should knowfreq 48%

basics

~20 s

Make the destination a parameter: accept a pathlib.Path argument, defaulting to the production location. The test then passes a path inside a temporary directory, drives the real write-and-rollback code, and inspects the files that survive.

open as a page

A loader module reads os.environ at import time; why does patching the variable in the test change nothing?

level: seniorimportance: should knowfreq 40%

basics

~20 s

The module body ran once, at first import, and froze the value into a module global; every later import returns that same cached module. A change to os.environ afterwards cannot travel back into an expression that already evaluated.

open as a page

Why does joining a Python set of stop ids give a route-optimisation job a different cache key each run?

level: seniorimportance: should knowfreq 38%

basics

~20 s

CPython randomises the hash of str and bytes per process, and a set stores entries by hash, so its iteration order differs between processes. Any key built from that order differs too. Join sorted(ids) instead.

open as a page

How do you choose across a Python codebase between hand-written fakes, unittest.mock.patch, and the real dependency?

level: principalimportance: should knowfreq 35%

basics

~20 s

Decide by ownership and by what the test is for: own the interface, own a fake shipped beside the implementation; patch only edges you cannot change; use the real dependency where fidelity is the point.

open as a page

How does typing.Protocol let a hand-written fake stand in for a real collaborator?

level: middleimportance: nice to knowfreq 25%

basics

~20 s

Declare a Protocol naming only the methods your code calls, then annotate the injected parameter with it. Conformance is structural, so a fake matches by having those methods - no base class - and a static checker verifies both sides.

open as a page

Where does io.StringIO stop being a faithful stand-in for a file opened by open()?

level: middleimportance: nice to knowfreq 28%

basics

~20 s

io.StringIO holds str only, so no encoding or decoding ever happens; its default newline setting does not translate CRLF the way open() does; and it has no operating-system descriptor, so fileno() raises io.UnsupportedOperation. Byte-level code needs io.BytesIO.

open as a page

What are the risks of installing a stub module into sys.modules during a test?

level: seniorimportance: nice to knowfreq 20%

basics

~20 s

sys.modules is one process-wide import cache, so an entry you add is what every later import of that name returns — in any module and in every test that follows. Nothing removes it for you.

open as a page