Cardinality & Key Size

Sizing a key scheme is arithmetic, and the reason it is worth doing on paper is that the store will never do it for you: it accepts each write on its own merits and the design is visible only in the total. ## Step one: multiply the template out Read the key template as a product of factors. "One entry per user per day per field" is three factors, and each one is a number you can obtain from someone: - **users** — from the business, plus a growth assumption you state out loud; - **days of history retained** — from the requirement, not from the code, and often nobody has decided it; - **fields per user per day** — from the design; this one is fixed and small. The product is the standing **distinct-key count**: how many separate keys exist at any moment once the scheme reaches steady state. Keep it separate from the *rate*, which is keys added per day — you need both, because the rate tells you how fast a wrong assumption becomes an incident. ## Step two: price one entry, honestly The bytes you chose are the key string plus the value. Both are real: the key is stored, not metadata that lives somewhere free. But the bytes you chose are a floor, not the answer. Every entry in a store of this class carries fixed bookkeeping beyond its own bytes — how much varies between stores, and by enough that a figure carried over from one store is worthless on another. The way to get it is to measure: write a representative sample of real entries at production sizes into a tier you can inspect, and read the difference. That per-entry cost is its own subject; for the arithmetic here, treat it as a measured constant you add. ## Step three: find the unbounded factor | Factor | Bounded by | What it does over time | |---|---|---| | users | the business | grows, slowly and predictably | | fields per user per day | the design | fixed unless the design changes | | days retained | nothing, by default | grows forever until something removes entries | The third row is where most of these schemes fail. A per-day key is a new key every day, permanently, unless each entry carries a deadline after which the store removes it, or a job removes them. The lifetime is the other lever available here and how those deadlines behave is a separate subject; the arithmetic point is simply that the days factor is a decision someone has to make, and if nobody makes it the factor is infinity. ## Step four: consider the other shape, with the precondition attached The count is a consequence of how many keys the design mints, so the sharpest lever is to mint fewer. Folding the four per-day fields under one key — a single entry per user per day, holding the fields inside it — divides the distinct-key count by four and removes three copies of the key text. That move has a precondition that is easy to assert and wrong on much of this class of store. It is worth having only where the server understands the value as a structure and can read or change part of it; on a store where a value is **opaque bytes the server only hands back**, one entry holding four fields means every change to any field is a full read, modify and write of the whole value, and every read of one field pulls all four across the network. What the structure can do is another subject entirely — here it is only a premise that decides whether the folding is a saving or a tax. ## Step five: state the answer as a range with assumptions The output of this exercise is not one number. It is a figure, the assumptions it rests on, and the factor that will invalidate it: 1. "At two million users, ninety days retained and four fields, the scheme stands at seven hundred and twenty million entries." 2. "At the bytes we chose, that is roughly forty gigabytes before per-entry bookkeeping, which we measured at our sizes and added." 3. "Days retained is the unbounded factor, and today nothing bounds it." 4. "At five million users the same scheme is two and a half times this figure, and the answer is a different design, not more memory." An engineer who can produce those four lines before shipping has done the work this subject exists for. ## The whole method: multiply the factors, then price the bytes you chose. The figure counts chosen bytes only; the fixed bookkeeping each entry carries is measured on your own tier and added on top ``` users 2,000,000 days of history retained 90 fields per user per day 4 --------------------------------------------- distinct-key count 2,000,000 x 90 x 4 = 720,000,000 entries key bytes (template plus identifiers) ~48 value bytes (a short number) ~8 --------------------------------------------- chosen bytes 720,000,000 x 56 bytes = about 40 gigabytes keys added per day 2,000,000 x 4 = 8,000,000 unbounded factor days retained ```