skip to content

A collector's os.walk over a spool directory silently skips subtrees and hits missing files. Why?

level: seniorimportance: should knowfreq 35%

answer

  1. Lazy, one directory at a time
  2. Never a consistent snapshot of the tree
  3. The default error policy is silence
  4. Pruning needs a slice assignment
  5. Symlink cycles have no visited set

basics

~20 s

os.walk lists one directory at a time as you consume it, so a tree changing underneath is never a consistent snapshot — and by default it ignores listing errors, so an unreadable subtree looks empty instead of raising.

solid answer

~50 s

`os.walk` is a generator that lists one directory at a time as you consume it, so there is no snapshot of the tree: a file can be rotated away between being listed and being opened, which surfaces as `FileNotFoundError` at open time and is handled by catching it, not by checking `os.path.exists` first. Its `onerror` parameter defaults to `None`, which means errors raised while listing a directory — permission denied most often — are **ignored**, so a subtree you cannot read yields nothing and looks empty; pass a callable that logs or re-raises. With the default `topdown=True` you can prune the traversal by mutating `dirnames` **in place** (`dirnames[:] = [...]`); rebinding the name does nothing. `followlinks` is `False` by default, and turning it on can loop forever on a symlink cycle. Finally the walk yields bare names, so join them with `os.path.join(dirpath, name)`.

code

python · 16 lines
python
import os


def on_error(err: OSError) -> None:
    print(f"cannot list {err.filename}: {err}")


def walk_spool(root: str) -> list[str]:
    found: list[str] = []
    for dirpath, dirnames, filenames in os.walk(root, onerror=on_error):
        dirnames[:] = [d for d in dirnames if d not in {"quarantine", ".git"}]
        found.extend(os.path.join(dirpath, name) for name in filenames)
    return found


print(len(walk_spool(os.getcwd())))

go deeper

for a junior

Know that os.walk yields (dirpath, dirnames, filenames), that the last two are bare names needing os.path.join, and that it descends the whole tree by default.

for a middle

Explain the mechanics: it is a generator listing one directory at a time, topdown=True lets you prune by mutating dirnames in place, followlinks is off by default, and onerror decides what happens when a listing fails.

for a senior

Demonstrate the production judgment: a tree changing underneath the walk is normal, so handle failures at the point of use rather than pre-checking, always pass an error handler so an unreadable subtree is not silently empty, and key any bookkeeping on something more stable than a filename.

for a principal

Own the design question — whether scanning a directory tree is the right integration at all, versus an explicit hand-off protocol such as atomic rename into a ready directory, and what that choice costs in throughput, duplicate work and operator visibility.

## What os.walk actually does `os.walk(top)` returns a generator. Each time you advance it, it lists exactly one directory — internally using the scandir mechanism, which is why entries arrive with type information already attached and the walk is far cheaper than a listing plus a stat call per entry — and yields a three-tuple `(dirpath, dirnames, filenames)`. `dirpath` is a path; `dirnames` and `filenames` are **bare names** within it, so any real use joins them with `os.path.join`. Nothing about that is atomic. The tree is read incrementally, over however long your loop body takes, while the rest of the machine keeps changing it. ## Symptom one: files that vanish between listing and use A telemetry collector walking a spool directory while a rotation job renames finished files and deletes ingested ones is the canonical case. The name arrived in `filenames`; by the time the loop body opens it, it is gone. There is no way to prevent this — a check-then-use pattern (`if os.path.exists(p): open(p)`) narrows the window and does not close it, and it is slower. The correct shape is to attempt the operation and handle the failure: ```python try: with open(path, "rb") as handle: ingest(handle) except FileNotFoundError: continue # rotated away underneath us; not an error except PermissionError: log_and_skip(path) ``` That is the ask-forgiveness idiom applied to a race you do not own. The corollary is that the collector's own bookkeeping must be keyed on something stable — a content identifier or a record identifier inside the file — rather than on a path. A collector that dedupes by filename and reports an 83% hit rate against its in-memory index of ingested files is reporting a number about *names*, and rotation renames files, so every rotation quietly turns already-ingested data into a cache miss and a duplicate ingest. ## Symptom two: subtrees that disappear without a word This is the one that surprises people. `os.walk` takes an `onerror` parameter, and it defaults to `None`, meaning **errors are ignored**. When listing a directory fails — most commonly a permissions problem, sometimes a directory deleted mid-walk — the walk does not raise and does not warn. It simply yields nothing for that subtree and moves on. A collector missing an entire sensor's worth of files because one directory was created with restrictive permissions will show no error anywhere. Always pass a handler: ```python def on_error(err: OSError) -> None: logger.warning("cannot list %s: %s", err.filename, err) for dirpath, dirnames, filenames in os.walk(root, onerror=on_error): ... ``` The callable receives the `OSError` instance, whose `filename` attribute names the directory that failed. If a missing subtree should be fatal, re-raise from inside the handler. ## Pruning: in place, or not at all With the default `topdown=True`, each tuple is yielded *before* its subdirectories are visited, and the walk consults the `dirnames` list afterwards to decide where to descend. So you can prune — skip a quarantine directory, skip a hidden tree, skip anything already processed — by mutating that list: ```python dirnames[:] = [d for d in dirnames if d != "quarantine"] ``` The slice assignment is load-bearing. `dirnames = [...]` rebinds the local name and the walk never sees it, and this is the single most common `os.walk` bug in review. With `topdown=False`, children are yielded before their parent, which is what you want when removing a tree bottom-up — but pruning does not work at all in that mode, because the descent has already happened. ## Symlinks and cycles `followlinks` defaults to `False`: symlinks to directories are reported in `dirnames` but not descended into. Setting it to `True` makes the walk follow them, and a symlink pointing at an ancestor then produces an infinite traversal, because `os.walk` keeps no record of what it has visited. If you must follow links, track visited directories by device and inode number yourself. ## When os.walk is the wrong tool Two alternatives are worth naming in an interview. For a single directory you do not need a walk at all — the scandir call gives you entries with cached type information and is the cheapest option. And when the race matters for correctness rather than merely for error handling — you must be sure the file you open is the file you listed, in a tree an untrusted process can rearrange — the file-descriptor-based walk that the `os` module also provides yields a directory descriptor for each level, so operations can be performed relative to that descriptor instead of re-resolving a path that may have been swapped underneath you. ## The four-sentence answer `os.walk` is lazy and per-directory, so it is never a snapshot. Its default error policy is silence, so pass `onerror`. Prune with an in-place mutation of `dirnames` under the default top-down order. And handle disappearing files at the point of use rather than trying to check first.

  • How do you skip a subdirectory during an os.walk, and what is the classic mistake?
    Mutate the `dirnames` list in place — `dirnames[:] = [d for d in dirnames if keep(d)]` — because the walk re-reads that same list object to decide where to descend. The classic mistake is `dirnames = [...]`, which rebinds a local name the walk never sees, so the pruning silently does nothing. It also only works with the default top-down order; bottom-up has already descended by the time you get the tuple.
  • The rotation job renames each finished file into place. Should it use os.rename or os.replace?
    `os.replace` — it overwrites an existing destination atomically on every platform, whereas `os.rename` raises `FileExistsError` if the target exists on Windows. Both are atomic only within a single filesystem; across filesystems the operation raises `OSError` and you need a copy-then-replace. Renaming a fully written temporary file into place is the standard way to make a reader see either the old file or the complete new one, never a partial write.
  • Why not check os.path.exists before opening each file the walk reported?
    Because the check and the open are two separate operations with a window between them, so the file can still vanish — it converts a guaranteed failure into an intermittent one while costing an extra system call per file. Attempt the open and catch `FileNotFoundError`. The same reasoning is why file-descriptor-relative operations exist for the cases where the identity of what you opened actually matters.

  • Believes os.walk takes a consistent snapshot of the tree
  • Assumes a listing failure raises rather than being ignored
  • Prunes by rebinding dirnames instead of slice assignment
  • Checks existence before opening to avoid a race
  • Thinks os.walk follows symlinked directories by default
  • Treats the yielded names as full paths

os.walk has been implemented on top of the scandir mechanism since Python 3.5, which is why entries carry type information and the traversal is cheap; the onerror, topdown and followlinks semantics described here are unchanged and hold on CPython 3.14.

context