A collector's os.walk over a spool directory silently skips subtrees and hits missing files. Why?

## What os.walk actually does `os.walk(top)` returns a generator. Each time you advance it, it lists exactly one directory — internally using the scandir mechanism, which is why entries arrive with type information already attached and the walk is far cheaper than a listing plus a stat call per entry — and yields a three-tuple `(dirpath, dirnames, filenames)`. `dirpath` is a path; `dirnames` and `filenames` are **bare names** within it, so any real use joins them with `os.path.join`. Nothing about that is atomic. The tree is read incrementally, over however long your loop body takes, while the rest of the machine keeps changing it. ## Symptom one: files that vanish between listing and use A telemetry collector walking a spool directory while a rotation job renames finished files and deletes ingested ones is the canonical case. The name arrived in `filenames`; by the time the loop body opens it, it is gone. There is no way to prevent this — a check-then-use pattern (`if os.path.exists(p): open(p)`) narrows the window and does not close it, and it is slower. The correct shape is to attempt the operation and handle the failure: ```python try: with open(path, "rb") as handle: ingest(handle) except FileNotFoundError: continue # rotated away underneath us; not an error except PermissionError: log_and_skip(path) ``` That is the ask-forgiveness idiom applied to a race you do not own. The corollary is that the collector's own bookkeeping must be keyed on something stable — a content identifier or a record identifier inside the file — rather than on a path. A collector that dedupes by filename and reports an 83% hit rate against its in-memory index of ingested files is reporting a number about *names*, and rotation renames files, so every rotation quietly turns already-ingested data into a cache miss and a duplicate ingest. ## Symptom two: subtrees that disappear without a word This is the one that surprises people. `os.walk` takes an `onerror` parameter, and it defaults to `None`, meaning **errors are ignored**. When listing a directory fails — most commonly a permissions problem, sometimes a directory deleted mid-walk — the walk does not raise and does not warn. It simply yields nothing for that subtree and moves on. A collector missing an entire sensor's worth of files because one directory was created with restrictive permissions will show no error anywhere. Always pass a handler: ```python def on_error(err: OSError) -> None: logger.warning("cannot list %s: %s", err.filename, err) for dirpath, dirnames, filenames in os.walk(root, onerror=on_error): ... ``` The callable receives the `OSError` instance, whose `filename` attribute names the directory that failed. If a missing subtree should be fatal, re-raise from inside the handler. ## Pruning: in place, or not at all With the default `topdown=True`, each tuple is yielded *before* its subdirectories are visited, and the walk consults the `dirnames` list afterwards to decide where to descend. So you can prune — skip a quarantine directory, skip a hidden tree, skip anything already processed — by mutating that list: ```python dirnames[:] = [d for d in dirnames if d != "quarantine"] ``` The slice assignment is load-bearing. `dirnames = [...]` rebinds the local name and the walk never sees it, and this is the single most common `os.walk` bug in review. With `topdown=False`, children are yielded before their parent, which is what you want when removing a tree bottom-up — but pruning does not work at all in that mode, because the descent has already happened. ## Symlinks and cycles `followlinks` defaults to `False`: symlinks to directories are reported in `dirnames` but not descended into. Setting it to `True` makes the walk follow them, and a symlink pointing at an ancestor then produces an infinite traversal, because `os.walk` keeps no record of what it has visited. If you must follow links, track visited directories by device and inode number yourself. ## When os.walk is the wrong tool Two alternatives are worth naming in an interview. For a single directory you do not need a walk at all — the scandir call gives you entries with cached type information and is the cheapest option. And when the race matters for correctness rather than merely for error handling — you must be sure the file you open is the file you listed, in a tree an untrusted process can rearrange — the file-descriptor-based walk that the `os` module also provides yields a directory descriptor for each level, so operations can be performed relative to that descriptor instead of re-resolving a path that may have been swapped underneath you. ## The four-sentence answer `os.walk` is lazy and per-directory, so it is never a snapshot. Its default error policy is silence, so pass `onerror`. Prune with an in-place mutation of `dirnames` under the default top-down order. And handle disappearing files at the point of use rather than trying to check first.