Memory as the Medium

## Two components, not one dial An **in-memory store** keeps the dataset it is holding in the memory of the process that serves it. Answering a read means addressing the entry and handing back the bytes; for the entries it holds resident there is no step in that path that reaches a storage device. The store therefore makes a **residency guarantee**: what it currently holds, it holds in RAM. A **disk engine** makes a weaker claim. Its data lives in files, and it keeps a large in-memory area of recently used pages. A read whose page is already in that area is also a memory read, and costs the server roughly what the in-memory store's read costs. A read whose page is not there pays a device access first. The engine has two latencies, and which one a request gets depends on what has been touched recently and on how much memory the engine was given. That is the real distinction, and it is not the slogan: **guaranteed residency against probable residency**. In a healthy, well-provisioned system both components answer from memory most of the time. ## Where the difference actually appears | what is read | rough cost of the access | what governs it | |---|---|---| | an entry already in RAM | around a hundred nanoseconds | memory latency | | a page on a solid-state device | tens to a few hundred microseconds | device access latency | | a page on a spinning disk | several milliseconds | seek and rotation | | anything across a datacentre network | a few hundred microseconds | the hop, whatever the medium | The figures are rough and move with every hardware generation; the **ordering** is the stable part. Read the table as the shape of the comparison, not as numbers to quote back. Now put a workload through it. Suppose an engine answers ninety-nine reads in a hundred from resident pages. Its **median** read is a memory read, hard to tell apart at the server from the in-memory store's. Its **high percentiles** are a different animal: the hundredth read pays a device access two to four orders of magnitude slower than the median. The in-memory store's distribution is narrower, because for resident entries it has no miss to have. So what the medium buys is best stated as the shape of the latency distribution: - the median is often similar once the engine is warm; - the tail is where the residency guarantee shows; - the tail is what a request timeout, a queue depth or a high-percentile objective is actually made of; - neither component escapes the network: if it is reached over a hop, the hop is usually larger than the medium difference being argued about. ## What the guarantee costs RAM is the most expensive storage in the machine, and a machine takes only so many modules. A component that promises residency is therefore promising that what it holds fits in memory somebody paid for. That bill is the other half of the medium story, and it is why an in-memory tier normally holds a chosen subset of what the system knows rather than all of it. How large that subset should be, what one entry costs beyond its payload, and what the store does once it fills are separate subjects with their own answers; the medium only forces the question to exist. ## Where stores in this class differ None of the above should be stated as though every store behaved identically: - **Not every store keeps everything in RAM.** Some in this class can keep colder values on a solid-state device and fetch them back on access. That trades part of the residency guarantee for capacity, and the tail comes back with it. - **Placement changes the arithmetic.** The component may be reached over a network or may sit in the application process. Only the second avoids a hop, and the hop dominates most medium differences. - **Engines differ enormously in residency.** One deployment never misses its engine's memory; another misses constantly. Any claim about the engine side of this comparison is a claim about a particular deployment, not about disk. ## Saying it in an interview Lead with what each component promises rather than with a ratio: the store guarantees a memory read, so its distribution is narrow and its capacity is bounded by what RAM costs; the engine usually answers from memory too, and its interesting number is the tail rather than the median. Then name the baseline before quoting any figure - which device, warm or cold, and whether a network hop sits in the path. Whether a given design should own such a component at all is a separate question, decided on failure domains and measured evidence, not on the medium alone.