A sampling pass names the ten largest entries in a store, but they explain little of the stored-data size; where is the memory?

## Two questions that look like one Operationally there are two distinct findings, and confusing them is the most common way an attribution exercise ends with nothing actionable. - **"Is one entry big enough to hurt other callers?"** A single very large entry is an operational hazard in its own right: it is expensive to transfer, expensive to remove, and on a server that executes one operation at a time it delays everybody while it is handled. Whether an oversized entry is a design fault, and what to do about it, is its own subject — here it is only something the walk can find. - **"Which prefix is holding the memory?"** This is an accounting question, and the answer is a sum. It has no relationship to the first. A **sampling pass over entries** that reports the largest entries it saw answers the first question. It is blind, by construction, to the second. ## Why a top-N is blind to a heavy prefix The arithmetic is the whole insight. Suppose the tier holds: - ten entries of one megabyte each — about ten megabytes, and every one of them lands in a top-ten list; - six million entries of two kilobytes each under a single prefix — about twelve gigabytes, and not one of them is ever ranked. The list is correct and useless. The memory is a *distribution*, and a ranking of individuals cannot see a population. The same blindness runs the other way: a prefix that is light in total can still contain the one entry that stalls a caller, so a rollup alone does not replace the top-N either. Both findings are needed, and they answer different pagers. ## The accumulator that answers both During a single walk, per prefix, keep: 1. a **running sum of estimated bytes** — who holds the memory; 2. a **count of entries** — whether growth is population or size; 3. the **largest entry seen**, with its key — the outlier hunt, for free, on the same pass. The derived average (bytes divided by count) is what makes the report readable: two prefixes at the same total behave completely differently at a hundred bytes each and at ten megabytes each, and the remedies have nothing in common. | Finding | Answers | Blind to | |---|---|---| | Largest individual entries | whether one entry can stall a caller or a transfer | a prefix made of millions of small entries | | Heaviest prefixes by summed bytes | who is holding the memory | one rare outlier inside an otherwise light prefix | | Entry count per prefix | whether a population or an entry size is growing | skew hidden inside a single prefix | ## What sampling does to each finding Sampling is what makes the pass affordable, and it treats the two findings very differently: - **Per-prefix sums survive sampling well.** A sample drawn across the keyspace lands in a prefix roughly in proportion to that prefix's share of the entries, so scaling by the sampling rate converges quickly on the shape — which prefixes dominate, and by roughly what factor. - **Outliers survive sampling badly.** A single enormous entry is one member out of millions and may never be drawn. If outliers are genuinely the point, either walk exhaustively, or check a size threshold on every key as you pass it and record only the exceedances, which is cheap. Quote a scaled sample as an estimate with its rate attached. A per-prefix figure presented to three significant figures from a one-in-a-thousand sample is a claim the measurement cannot support. ## Reporting it so the two are not confused Publish them as two tables under two headings, never as one merged list. State the walk's window and whether it was full or sampled. Where the store could not report per-entry sizes and you measured fetched values instead, say so, because those numbers exclude whatever the entry costs beyond its bytes and are not comparable to the server's own total. The residual gap between your summed estimates and that total is itself worth printing: it is the honest error bar on the whole exercise, and a reader who sees it will trust the rest.