The Round Trip

## Two numbers, not one A single read against an in-memory store costs the caller two things that scale for entirely different reasons. - **The server's service time** — what the store spends executing the operation once the request is in its hands. For a single-key read of a small value this is a hash lookup and a copy into an outbound buffer: microseconds, sometimes a fraction of one. - **The caller's wall time** — what the caller measures from issuing the call to holding the value: the service time, the network journey out and back, and the encoding and decoding at both ends. On any deployment where the store is not on the caller's own host, the second number is one to three orders of magnitude larger than the first, and everything here follows from that ratio: **you are not designing against the store's speed, you are designing against the number of times you cross the network.** ## Why a loop multiplies only the network term "Sequential" has a precise meaning on the wire: the caller cannot send call number two until reply number one has arrived — because the next key depends on what came back, or simply because the code is written as a loop. The calls cannot overlap, so the request's wall time is roughly > trip count × (one round-trip time + service time) Service time is negligible inside that bracket, so it reduces to **trip count × round-trip time**. Five hundred reads against a store answering in 5 microseconds spend about 2.5 ms inside the store, and the remaining 297 ms waiting for the network, five hundred separate times. The same loop is invisible at one placement and fatal at another, so redo the arithmetic at each. Orders of magnitude only — measure your own path: | Placement of the tier | One round trip | 500 sequential reads | The same 500 keys in one operation | |---|---|---|---| | Same host, over a local socket | tens of microseconds | tens of milliseconds | well under a millisecond | | Same zone, one network hop | a few hundred microseconds | a few hundred milliseconds | roughly a millisecond | | Different zone, same region | around a millisecond | half a second or worse | a few milliseconds | | Different region | tens of milliseconds | many seconds — a design error | tens of milliseconds | The right-hand column is the point. Collapsing the crossings changes the answer by two or three orders of magnitude; making the store itself twice as fast changes it by nothing anyone could measure. ## Why every server-side number stays green This is the failure mode's signature, and the reason it survives so long in production: 1. **The store's own accounting is honest and useless here.** It reports service time, which really is microseconds, and nothing in it counts how many times one request came back for more. 2. **The throughput ceiling is nowhere near.** Five hundred operations spread over one request is nothing against a tier answering hundreds of thousands per second, so utilisation looks idle. 3. **Per-call latency measured at the caller also looks fine.** Each call genuinely is fast. The cost lives in the count, and no per-call statistic — mean or high percentile — has a count in it. The cost becomes visible only where someone measures **the caller's wall time for the whole request path alongside the number of calls that path made.** A pair of numbers, never one. ## The shape it makes in a request trace Drawn on a timeline, the pattern is recognisable before you read a single label: a long row of very narrow calls, all about the same width, none overlapping another, together filling most of the request. Each attribute carries information. - **Uniform width** says the work per call is constant — no single call is doing anything unusual. - **Strict sequencing with no overlap** says nothing was sent before the previous reply landed. - **The count** is the cost, and it is the only thing on that row that is large. Two neighbouring shapes look nothing like it: one genuinely expensive operation is a single wide bar, and contention shows as varying widths separated by gaps. ## The directions out All of them do the same thing — remove crossings — and none makes the store faster: - **Ask for fewer keys.** The cheapest crossing is the one the path never needed. - **Ask for many keys in one operation**, where the store offers such an operation and the keys can be served together. - **Send a run of operations without waiting for each reply**, so the waiting happens once instead of once per call. - **Move the decision to where the data is**, so the caller asks one question rather than fetching the inputs to answer it itself. - **Move the tier closer**, shrinking every crossing rather than removing any. ## What varies between stores The arithmetic is universal; its inputs are not. Some stores in this class offer an operation taking many keys and some answer only single-key reads and writes, which decides whether the second direction exists at all. Some execute operations one at a time while others serve requests from a thread pool — that distinction changes who is delayed behind an expensive call, but not this arithmetic, because the term being multiplied is negligible under either model. And where the keyspace is split across nodes, one operation over many keys may not be servable by a single node, so the honest count is the number of nodes involved rather than one.