Why use hashlib.file_digest to hash a 4 GB file instead of reading it first?

### The problem it replaces Before Python 3.11 every codebase carried its own version of this loop: ```python h = hashlib.sha256() with open(path, "rb") as f: while chunk := f.read(65536): h.update(chunk) ``` and a depressing number of codebases carried the shortcut instead: ```python h = hashlib.sha256(open(path, "rb").read()) ``` The shortcut is correct for a 40 KB config and catastrophic for a 4 GB export: `read()` builds one `bytes` object holding the whole file, so peak resident memory tracks the largest input you will ever be handed. On a container with a memory limit it does not degrade — it dies, and it dies on the one day someone uploads an unusually large batch. `hashlib.file_digest(fileobj, digest)` is that loop, written once, in the standard library. ### What it actually does The second argument is either an algorithm name (`"sha256"`) or a callable returning a hash object (`hashlib.sha256`), so it composes with `hashlib.new()`-style configuration. Then: * If the object exposes `getbuffer()` — an `io.BytesIO`, for instance — it hashes that buffer directly, with no copy at all. * Otherwise it requires a readable binary file. It allocates a single `bytearray` of 256 KiB, wraps it in a `memoryview`, and loops on `readinto(buf)`, feeding `view[:size]` to `update()` until a read returns 0. Two details make that better than the naive loop. `readinto()` fills an existing buffer rather than allocating a fresh `bytes` per chunk, so the allocator sees one buffer for the whole file instead of tens of thousands of short-lived objects. And the `memoryview` slice hands the hash a view of the filled prefix without copying the tail. It returns the **hash object**, not a digest — a small API decision people trip over exactly once: ```python with open(path, "rb") as f: digestobj = hashlib.file_digest(f, "sha256") print(digestobj.hexdigest()) ``` Returning the object means you can also ask for `digest()`, `digest_size` or `name` without re-reading the file. ### The constraints worth knowing **Binary mode is mandatory.** A file opened in text mode is a decoding wrapper with no `readinto()`, and the call raises `ValueError` naming the object as not being a file-like object in binary reading mode. That refusal is a feature: hashing decoded text would silently make the digest depend on the platform's default encoding and on newline translation, so the same file hashed on two hosts could disagree. **It reads from the current position to EOF and does not seek back.** If you have already consumed part of the stream, you hash only the remainder; if you need the digest and then the contents, seek to 0 yourself. This also makes it usable on a pipe or socket wrapper, where seeking is not an option at all. **It is a hashing helper, not a file API.** It does not open the file, does not close it, and does not care about the path — which is what lets it hash anything readable and binary, including an in-memory buffer, a decompressing stream, or a network body reader. ### Where it fits in real work Content addressing and change detection are the everyday uses: store `hexdigest()` alongside an artefact and you can answer "has this changed?" without a byte-by-byte comparison, and you can dedupe identical uploads that arrived under different names. For an integrity check against a published checksum, the digest must be compared to a value delivered over a channel you trust — the digest itself carries no key and an attacker who can rewrite the file can usually rewrite a checksum sitting next to it. Throughput is respectable because the hashing work happens in C and the interpreter releases the GIL around a sufficiently large buffer, so a thread pool hashing several files at once genuinely overlaps. The remaining cost is I/O, and the 256 KiB default is already tuned for it; the buffer size is a private parameter, not a public tuning knob, so if you truly need different chunking you write the loop by hand. On Python 3.10 and earlier there is no `file_digest` — write the `read()`/`update()` loop, and keep the chunk size in the hundreds of kilobytes rather than a few bytes. ### When you need more than a digest `file_digest` is for the case where hashing is the *only* job: it consumes the stream to EOF and you get a digest back. When you must also process the bytes — parse rows, write them elsewhere, compress them — go back to the explicit loop and call `update()` on each chunk as it passes through, so the data is read exactly once and the digest falls out for free at the end. Hashing on write is the same pattern in the other direction: update the digest as you emit bytes, then record it beside the artefact so a later reader can confirm what it received matches what was produced.