Why can open(path, 'w') expose an empty or half-written file to readers?

## Truncation happens at open, not at write When Python evaluates `open(path, "w")` it asks the operating system to open that name for writing **and to truncate whatever is already there to zero bytes**. That truncation happens as part of the open call - before your first `write()`, before you have serialized anything, and even if the program raises immediately afterwards and never writes at all. From that instant the old contents are gone. There is no shadow copy, no undo and no transaction to roll back. ## The new bytes arrive afterwards, in pieces What follows is not one step either. A text file object accumulates your writes in a userspace buffer and hands them to the operating system when that buffer fills, when you call `flush()`, or when the file is closed. So the sequence of states an outside observer can catch is: the full old contents, then an empty file, then a growing prefix of the new contents, then finally the complete new file. For a small file those states pass in microseconds; for a few megabytes on a busy volume the window is long enough to hit reliably. ## Why anyone can observe those states at all A path is a directory entry pointing at a file. Writing in place mutates that same file, so every process that opens the path is looking at the bytes you are currently rewriting. Unix applies no implicit locking: an open for writing does not exclude readers, and Python adds none of its own. Nothing anywhere in the stack promises that a reader sees either the old version or the new version and nothing between. ## What that looks like in production A route-optimisation job recomputes a 17-service dependency graph every few minutes and rewrites `routes.json` in place, while a supervisor process reloads the file whenever its modification time changes. Most of the time the reload lands between rewrites and everything is fine. Occasionally it lands inside the window and decodes an empty file, or an object cut off mid-key, and the loader raises. The failure is timing-dependent, so it reproduces on a loaded machine roughly once a week and never in a test. A crash makes the same window permanent. If the process is killed between the truncation and the final flush - an out-of-memory kill, a deploy, a power loss - what remains on disk after the restart is the truncated file. Overwriting in place converts a transient race into durable corruption, and the old good copy is not recoverable because it was destroyed at open time. ## Things that do not fix it * **Writing everything in one `write()` call.** The buffer still flushes on its own schedule, and the operating system is free to split the transfer. * **Writing faster, or writing a smaller file.** That shrinks the window; it does not close it. * **Using `pathlib.Path.write_text`.** It is the same truncate-then-write sequence with a shorter spelling. * **Calling `flush()` more often.** That makes partial states visible sooner, not less often. * **Advisory locking.** It works only if every reader - including tools you did not write - cooperates, which they generally do not. ## What does fix it Never modify the destination in place. Write the new contents into a different file in the same directory, close it, and put it into position with `os.replace(tmp, path)`, which overwrites the destination by repointing the directory entry in a single step. A reader either opens the old file or the new one, and a crash leaves one whole version or the other. ## The neighbouring modes Mode `'a'` avoids the truncation - it keeps existing bytes and appends - but it does not make an update atomic; a reader can still catch a half-appended record. Mode `'x'` refuses to open a path that already exists, which is useful for create-once files but says nothing about replacing one. Only the write-elsewhere-then-rename approach gives readers an all-or-nothing view of a file you are replacing.