In a shared-memory multiprocessor, a value that thousands of threads only read costs almost nothing to access, while a single value that many cores write becomes a throughput ceiling even when the program uses no locks at all. Explain the asymmetry, and describe the techniques you would use to remove such a hot spot.

## Where the asymmetry comes from Cache coherence enforces, per line, *many readers or one writer*. That single rule creates the entire cost difference. **Reads of a clean line replicate.** Every core that reads it obtains its own copy in a shared state. Afterwards each core hits in its own L1 at a few cycles, with zero interconnect traffic. Add cores and nothing degrades: read-only data has essentially unlimited read parallelism. **Writes cannot replicate.** Before a store commits, the writing core must hold the line exclusively, which means invalidating every other copy and, if another core has modified it, transferring the current data. That is a round trip through the interconnect — tens to a couple of hundred cycles on one socket, considerably more across sockets. Worse, the requests are inherently ordered: the line has one owner at a time, so N writers form a queue. Throughput on that line tends toward one transfer per ownership-transfer latency regardless of how many cores you add, and per-core throughput therefore *falls* as N grows. This is why a shared counter incremented by every core is a scalability bug even with no lock, no blocking and no contention in the program's own terms. Read-modify-write operations make it sharper still, because they must hold ownership across the read and the write. The hardware provides the serialisation the code appears to avoid. ## Recognising it The signature is a throughput curve that flattens early and then declines with added threads, concentrated on one instruction or one field, with cache misses served from other cores rather than from DRAM. Latency percentiles widen, because a thread's store may wait behind several ownership transfers. Distinguish it from a bandwidth limit (traffic goes to memory, not core to core) and from a lock convoy (threads park and wake rather than spin on ownership). ## Techniques, in the order worth trying **1. Reduce the write rate.** The best write is the one that never happens. Metrics, statistics and progress counters rarely need per-event global updates: accumulate in a register or thread-local slot and merge every N events or every few milliseconds. This turns millions of contended writes into thousands. Sampling (count one in K events and multiply) is acceptable for many observability uses. **2. Shard the state.** Give each core or thread its own slot, each aligned to its own cache line, and combine on read. Writes become uncontended and scale linearly; reads become O(number of shards). This is the standard scalable-counter design, and it generalises: sharded queues, per-core free lists, per-core RNG state, striped hash tables. The cost is that a read is a summary rather than a snapshot of an instant, which is usually fine because with concurrent writers there was never a meaningful instant anyway. **3. Make the hot data read-mostly.** If a structure is read constantly and updated rarely, mutate a copy and publish it, so readers keep hitting replicated clean lines and only the rare publication costs an invalidation. This is the copy-on-write/immutable-snapshot idea, and it converts the expensive access pattern into the cheap one. **4. Partition ownership.** Assign the state to a single owning thread and have others send it work; the owner's writes are always to a line it already owns exclusively. Message passing costs a hand-off, but the hand-off can be batched, whereas ownership transfer per increment cannot. This is the structural reason single-writer designs and per-core data structures perform well. **5. Layout fixes last.** Padding and alignment remove *false* sharing — accidental collisions between unrelated variables. They do nothing for genuine contention on the same variable. Reach for them once the sharing is known to be accidental, and treat them as tuned constants that need re-measuring on different hardware, since line sizes and adjacent-line prefetching vary. ## What to avoid Do not assume replacing a lock with an atomic operation fixes the scaling problem; both funnel through the same line ownership, and under high contention an atomic retry loop can be worse because failed attempts still take ownership. Do not add threads to a write-contended hot spot expecting improvement — that is the one change guaranteed to make it worse. And do not tune layout before establishing whether the sharing is real or accidental; the two problems have entirely different fixes.