A payment reconciliation job shards accounts with hash(account_id) % 8 — why does it lose rows?

## The failure A reconciliation job partitions the day's transactions across eight workers by `shard = hash(account_id) % 8`, and each worker writes matched and unmatched rows into its own output file. Totals come out short, and rerunning produces a different set of missing accounts each time. The cause is that `hash()` of a `str` is salted with a secret drawn per interpreter process. Every process in the pipeline — each pool worker, the driver, tomorrow's rerun, the machine in the other region — computes a different hash for `'ACCT-4417'` and therefore a different shard number. A row emitted into shard 3 by yesterday's run is looked for in shard 6 by today's. Worse, a partial rerun re-partitions the records it has already written, so accounts are double-counted in one shard and absent from another. Nothing in the code is wrong to read; the invariant it relies on — that a given account always maps to the same bucket — is simply not one that `hash()` provides. ## Why it looked fine Two things mask it in development. Single-process runs are self-consistent. The salt is fixed for the life of the process, so a job that shards, writes and reconciles inside one interpreter never sees a mismatch. The bug appears only when a second process is involved. And the way children are created matters. A child created with the `fork` start method inherits the parent's memory, salt included, so parent and children agree on `hash()`. A child created with `spawn` or `forkserver` is a fresh interpreter with its own salt, so it does not. That distinction is why this class of bug has a habit of appearing after an upgrade or a platform move rather than at the moment the code is written: since Python 3.14 the default start method on Unix platforms other than macOS is `forkserver`, macOS and Windows have used `spawn` for years, and `fork` must now be asked for explicitly. Code that quietly depended on inherited salts on Linux changes behaviour on upgrade. ## Confirming it before you fix it Do not guess. Print `hash('ACCT-4417')` from two freshly launched subprocesses and show they differ; then print it from a pool worker and from the driver in the same run and show that too. That takes a minute and it distinguishes this cause from the other candidates — a partitioner that drops rows on a boundary, a worker that dies mid-write, or an output file that was never flushed. That last one deserves a note: when a shard worker is killed part-way, an output file left unclosed keeps its final buffered block unwritten, which produces short totals that look exactly like a partition bug. Rule it out by checking that every writer runs under a `with` block or an explicit close, so a crash cannot be confused with a mis-shard. ## The fix Derive the shard from a function that is stable everywhere. `hashlib.blake2b(account_id.encode('utf-8'), digest_size=8)` gives eight bytes you can turn into an integer with `int.from_bytes` and reduce modulo the shard count; it is fast, it is in the standard library, and it produces the same number on every interpreter, version and machine. `zlib.crc32` over the same encoded bytes is cheaper still and is fine where you need spread rather than collision resistance — worth knowing when the per-batch latency budget is tight and you are measuring at a high percentile rather than at the mean, because a per-row digest that is invisible in the average can still show up at the 92nd percentile of a large batch. Three details make the fix durable. Encode explicitly — `str` has no bytes until you choose an encoding, and hashing an implicitly encoded value is how the same account gets two shards on two locales. Pin the shard count somewhere the readers and the writer share, because changing it re-partitions everything just as surely as a changing salt does. And write the shard number into the output alongside the row, so a mismatch is detectable rather than silent. ## The general rule `hash()` exists to place objects in the in-memory `dict` and `set` implementations. Its contract is limited to one process: not stable across runs for text, not guaranteed across interpreter versions or word sizes even for the values that happen to be stable today, and not something an alternative implementation must reproduce. So the boundary is easy to state and easy to review for: any hash that outlives the process — a shard key, a cache key in a shared store, a filename, a partition column, a bloom filter written to disk, a deduplication fingerprint — comes from `hashlib` or a checksum, never from `hash()`. The same rule catches a subtler variant: `PYTHONHASHSEED` pinned in the deployment to 'fix' this. It papers over the symptom, ties correctness to how the process was launched, and re-opens the collision denial-of-service for any part of the service that parses untrusted keys. Fix the function, not the environment.