skip to content

What are the risks of installing a stub module into sys.modules during a test?

level: seniorimportance: nice to knowfreq 20%

answer

  1. The import cache is process-wide
  2. Every later import reads your entry
  3. Only imports after the stub see it
  4. Dotted names need their own entries
  5. patch.dict on the cache, or addCleanup

basics

~20 s

sys.modules is one process-wide import cache, so an entry you add is what every later import of that name returns — in any module and in every test that follows. Nothing removes it for you.

solid answer

~50 s

`sys.modules` maps module names to module objects, and the import statement consults it first, so `sys.modules["lab_vendor_client"] = stub` makes every subsequent import of that name anywhere in the process return your stub. That is the appeal: you can stand in for a module that is not installed in the test environment, is slow to import, or does real work at import. The risks follow from the same fact. Left behind, the entry outlives the test and silently serves the stub to everything after it. Code that already imported the real module keeps the object it bound, so the stub is invisible there. And a stub registered as a package name does not satisfy an import of its submodule. Install it with `patch.dict(sys.modules, {...})` or an `addCleanup` that deletes the key, and give the stub only the attributes the code actually uses.

code

python · 23 lines
python
import sys
import types
import unittest


class StubModuleTest(unittest.TestCase):
    def test_stub_is_removed_afterwards(self):
        stub = types.ModuleType("lab_vendor_client")
        stub.fetch_batch = lambda batch_id: {"rows": 6800}
        sys.modules["lab_vendor_client"] = stub

        def remove_stub():
            del sys.modules["lab_vendor_client"]

        self.addCleanup(remove_stub)

        import lab_vendor_client

        self.assertEqual(lab_vendor_client.fetch_batch("b-17")["rows"], 6800)


unittest.main(argv=["ignored"], exit=False, verbosity=0)
print("still cached:", "lab_vendor_client" in sys.modules)

go deeper

for a junior

Know that sys.modules is the import cache the import statement checks first, and that there is exactly one of it per process. You will rarely write to it, but you should recognise test code that does.

for a middle

Explain what an entry in sys.modules changes and what it cannot change — later imports yes, names already bound no — and how patch.dict or a registered cleanup guarantees the entry is removed.

for a senior

Show when the technique is warranted at all: a dependency that cannot be installed in the test environment, or one that does real work at import. Then show the cleanup discipline and a narrow stub that fails loudly on an unexpected attribute.

for a principal

Own the tradeoff between working around an untestable import and paying down the design that made it untestable, and set the rule for which external modules the suite is allowed to fake at all.

### What sys.modules is `sys.modules` is the interpreter's import cache: a plain mutable mapping from module name to module object, one per interpreter. The import statement checks it before doing anything else, and a successful import writes into it. Because it is a plain mapping, you can write into it yourself — and the import machinery will honour whatever you put there, without asking whether it came from a file. That gives tests a blunt but occasionally necessary tool. If the code under test does `import lab_vendor_client` at module scope and that distribution is not installed in the test environment, importing the code at all fails. Registering a `types.ModuleType` under that name before the import makes the code load and run against your object. ### The four risks **It is global, and it is not undone.** The entry is visible to every module in the process, not just the file that wrote it. Leave it behind and every later test that imports the name gets the stub — usually silently, sometimes producing a pass where a real failure was waiting. This is the same leak as an unrestored module attribute, with a wider blast radius, because it affects imports that have not happened yet. **Timing decides whether it works at all.** The stub only affects imports that occur *after* it is installed. If the module under test was imported during collection, it already bound the real object into its own namespace, and the import statement will never run again for it — your sys.modules entry sits there being consulted by nobody. This is why the technique belongs as close to the start of the process as it can be, and why it fits a genuinely unimportable dependency better than a merely inconvenient one. **Dotted names are separate entries.** Registering `lab_vendor` does not satisfy `import lab_vendor.results`. The machinery resolves each part of a dotted name and looks up the full name in the cache; on a miss it tries to import the submodule for real through the parent's submodule search path, which a plain module object does not have. If the code imports submodules, register every dotted name you need and also set each submodule as an attribute of its parent, because `from lab_vendor import results` reads it as an attribute. **A too-generous stub hides mistakes.** A bare `MagicMock` as the module answers every attribute access with a new mock, so a call to a function that does not exist on the real module passes the test and fails in production, and a renamed function is never noticed. Build the stub as a `types.ModuleType` with exactly the functions and constants the code under test uses, so an unexpected access raises `AttributeError` and the test tells you the truth. ### Doing it safely The cleanup must be structural, exactly as for any other shared state. `unittest.mock.patch.dict(sys.modules, {"lab_vendor_client": stub})` snapshots the mapping, applies the entry, and restores the snapshot on exit inside a `finally` — including the case where the real module was already cached and must be put back. `addCleanup` with a function that deletes the key is the equivalent when you need the stub installed for the duration of one test method rather than one block. What is never acceptable is a bare assignment with the removal written as the last line of the test. A second discipline is scope. Install the narrowest name that makes the import succeed, keep it in place for the shortest span that covers the import, and assert on the calls your stub received rather than on the stub's internals. ### When not to reach for it A stub module is a workaround for an import you do not control. When you do control the code, the cheaper fixes are ordinary: move the import inside the function that uses it so the module can be loaded without the dependency present, or accept the collaborator as an argument so no import needs to be intercepted at all. Reserve import-cache surgery for third-party or platform modules that genuinely cannot be imported in the test environment — a driver for hardware that is not attached, a platform-specific module on the wrong operating system, a heavy optional dependency your suite deliberately does not install. Stated as an interview answer: writing to `sys.modules` is legitimate, occasionally the only option, and always a promise to clean up — because you are editing a process-wide registry that every future import will read.

  • Your stub is in sys.modules but the module under test still uses the real dependency. Why?
    It imported the dependency before your stub existed and bound the resulting object into its own namespace. The import statement does not run again for it, so the cache entry is never consulted. Only imports that happen after the stub is installed see it. Either install the stub before the module under test is first imported, or work with the object that module is already holding.
  • Does registering a stub for a package name make an import of its submodule work?
    No. The machinery resolves each part of a dotted name and looks the full name up in the cache; missing it, it tries a real import through the parent's submodule search path, which a plain module object does not have. Register every dotted name the code imports, and set each submodule as an attribute of its parent so a from-import can find it.
  • Why is a bare MagicMock a poor stand-in for a whole module?
    It answers every attribute access with a new mock, so a call to a function that does not exist on the real module — a typo, or a name removed in a later release — passes the test and fails in production. A `types.ModuleType` carrying exactly the attributes the code uses raises AttributeError on anything unexpected, which is the feedback you wanted from the test in the first place.

saying these in an interview costs you the question

  • Thinks sys.modules is per test file or per module
  • Leaves the stub entry in place after the test
  • Expects a stub to affect modules that already imported
  • Assumes stubbing a package also stubs its submodules
  • Uses a bare mock so every attribute silently exists

context