After a cluster's connections were encrypted, readers replaying old history slowed far more than caught-up readers — why?

## Where the read speed came from in the first place On platforms whose records are appended to files and served from them, read throughput has an unusual property: the node barely participates. A reader asks for a range of stored records; the node identifies the region of the file that holds them and asks the operating system to send that region to the network connection. The bytes travel from the operating system's file cache out to the reader **without ever being copied into the serving process's own memory**. That is the **copy-free read path**, and it is why such a cluster can serve far more read bytes per second than its own processing would suggest. The path has a precondition that is easy to miss: the bytes leaving must be byte-for-byte the bytes stored. The moment they must be transformed on the way out, the process has to hold them. ## What an encrypted connection takes away - **The shortcut itself.** Encrypting output means transforming it, so the serving process must obtain every byte, transform it, and write the result. The stored region can no longer be handed to the network untouched. - **Extra copies and memory bandwidth.** Each byte now moves through the process rather than past it. On a node serving many gigabits, this is felt as processor time and memory traffic rather than as disk load. - **Per-connection setup work.** Establishing a protected connection costs more than establishing a plain one. Normally invisible, it becomes visible in a burst when a large client population reconnects at once. - **What it does *not* mainly cost: the arithmetic.** Current processors carry bulk-encryption instructions that make the transformation itself cheap per byte. A candidate who attributes the whole slowdown to "encryption maths" has the wrong model; the copying and the lost shortcut dominate. ## Why the bill lands on the history readers 1. **The added work is proportional to bytes shipped.** Not to connections, not to records, not to record age — to volume. 2. **A caught-up reader ships very little.** Its byte rate is bounded above by the write rate of what it follows, it usually asks for small recent ranges, and those ranges are frequently still in the operating system's file cache. It pays, but it pays at the rate the stream is being written. 3. **A replay is unbounded by comparison.** A reader rebuilding state from the start of retention, or a node rebuilding a copy from scratch after being replaced, pulls as fast as the cluster will let it. It is the single highest byte-rate flow in the cluster, and it is precisely the flow that used to be almost free. | Flow | Byte rate | Used the copy-free path | How the change is felt | |---|---|---|---| | Reader already caught up | Bounded by the write rate | Yes, on a small recent range | Slight, steady processor cost | | Reader replaying history | As fast as permitted | Yes, on large cold ranges | Large; the replay takes visibly longer | | Node rebuilding a copy | Highest in the cluster | Yes | The rebuild window stretches, so the cluster runs short of copies for longer | | Writer sending records | Bounded by the application | No — writes were never copy-free | Modest; writes always passed through the process | That last row is worth saying out loud in an interview: the write path never had the shortcut, so encryption does not change writes in the same way it changes reads. The asymmetry is the interesting part. ## This is a claim about a shape, not about every broker - On **log-shaped platforms serving reads from stored files**, the change is largest, because the shortcut existed and is now gone. - On a **queue-shaped broker that holds messages in memory and removes them once acknowledged**, there was never a copy-free read path to lose. The process already handled every byte, so the added cost is the per-byte transformation and the per-connection setup — real, but a different and usually smaller story. - On a **rented cluster**, member traffic is often encrypted already and the cost is inside the price. You will not see the difference; you also cannot choose it. Published figures for the throughput drop vary widely and depend entirely on record size, batch shape, how much history is being replayed and what the hardware is. Quoting one number as the answer is a mistake — the defensible claim is the mechanism and the direction. ## What to do with this in practice - Encrypt anyway, and budget for it: the answer to "it costs throughput" is capacity headroom, not leaving payloads in the clear. - Size the headroom against your **replay and rebuild** traffic, not against steady-state reads, because that is where the cost concentrates. - Expect a node rebuild to take longer once member traffic is protected, and account for that in how long the cluster tolerates running with fewer copies. - Measure on your own workload. A benchmark run on a different record size answers a different question.