skip to content

HyperLogLog

You will learn the probabilistic counter that estimates set cardinality in a fixed 12 KB, trading exactness for constant memory. Interviewers use 'count unique visitors at scale' to check you know when an approximate answer is the right engineering call.

part ofRedisoverview, primer and where to startread it →
on this pageshow

questions

4

You keep one Redis HyperLogLog per day of unique visitors. How do you produce a weekly and a monthly unique-visitor number, and what does the PFMERGE command guarantee about the accuracy of the result?

level: middleimportance: should knowfreq 40%

answer

  1. PFMERGE dest src... = per-register max
  2. union only: no intersect, no unmerge
  3. error does not compound when merging
  4. multi-key PFCOUNT unions on the fly
  5. cluster: same hash slot / hash tag

basics

~20 s

Union the daily keys: PFCOUNT day1 day2 ... estimates the union directly, or PFMERGE week day1 day2 ... stores a reusable weekly key. Merging takes the max of each register, so the merged key equals one built from all raw elements; error stays ~0.81% and does not accumulate across merges.

solid answer

~50 s

Daily HyperLogLogs roll up by union. Either call PFCOUNT with several keys, which computes the union on the fly, or PFMERGE weekly:2026-w33 uv:day1 ... uv:day7 to persist a weekly key you can then PFCOUNT cheaply and merge further into a month. The guarantee comes from how registers work: each register stores a maximum, so the union is a per-register max of the sources. The merged structure is exactly what you would have got by PFADDing every element of every day into one key. That means the ~0.81% standard error applies to the merged result too and does not compound as you merge weeks into months. It also means double counting is impossible: a visitor present on five days contributes once. The caveat is that this only works for unions. There is no intersection or subtraction, so 'visitors who came in both weeks' cannot be derived reliably.

code

text · 9 lines
text
PFADD uv:2026-08-10 u1 u2
PFADD uv:2026-08-11 u2 u3

PFCOUNT uv:2026-08-10 uv:2026-08-11   # union estimate, nothing stored
(integer) 3

PFMERGE uv:week:33 uv:2026-08-10 uv:2026-08-11
PFCOUNT uv:week:33
(integer) 3

go deeper

for a junior

Know that you union daily keys with PFMERGE or a multi-key PFCOUNT instead of adding the daily numbers together.

for a middle

Explain the per-register max, why that means no double counting and no extra error, and when you would persist a weekly key.

for a senior

Cover operational shape: key naming, TTLs on dailies, cost of merging many keys, cluster hash tags, and why intersections are off the table.

for a principal

Decide the rollup hierarchy and retention: which granularities are stored, which are computed on demand, and where an exact system must take over for questions HyperLogLogs cannot answer.

## Why rollups are the killer feature With exact Sets, a weekly unique count means SUNIONSTORE over seven large Sets: memory and CPU proportional to total members. With HyperLogLogs the same rollup costs a few kilobytes and constant time, which is why time-bucketed HyperLogLog keys are the standard shape for uniques analytics in Redis. ## The two ways to union PFCOUNT accepts multiple keys and returns the estimated cardinality of their union without storing anything for the union itself. PFMERGE dest src [src ...] computes the same union and stores it in dest, which may itself already exist and is then treated as another source. Persisting the weekly key matters when the daily keys expire, or when the monthly rollup should be cheap: merging four weekly keys is four register scans instead of thirty. ## Why the merge is lossless Each of the 16384 registers holds the largest leading-zero run observed for hashes that landed there. Maximum is associative, commutative and idempotent, so taking the per-register max of several structures produces exactly the structure you would have built by feeding all elements into a single key. This is the important property to state in an interview: merging introduces no additional error. A merged key carries the same ~0.81% standard error as any other key of that cardinality, whether you merged seven days or a hundred. Idempotence is what kills double counting. A visitor seen on Monday and Thursday sets the same register to the same value in both days' keys; the max of the two is that value, so the union counts them once. Summing daily PFCOUNT results instead would badly overcount every returning visitor. ## Practical shape Use deterministic key names such as uv:2026-08-14, set a TTL on the dailies, and run merges from a scheduled job. Because the merged key is just a Redis String, it survives DUMP/RESTORE and replication like any other value. If Redis runs in cluster mode, multi-key PF commands require all keys in the same hash slot, so give the family a hash tag such as uv:{uv}:2026-08-14 — otherwise the merge is rejected as a cross-slot operation. ## What you cannot do Only unions. There is no PFINTERSECT. Deriving an intersection through inclusion-exclusion (|A| + |B| - |A union B|) is mathematically valid but numerically terrible: you subtract two large approximate numbers, so the absolute errors of both survive while the result is small, and the relative error of the intersection can be enormous or even negative. If the product needs 'returning visitors', that question needs exact Sets, bitmaps over dense user IDs, or an offline system — not HyperLogLog arithmetic. Similarly you cannot subtract a day out of a merged key. Merged registers no longer remember where their maxima came from, so unmerging is impossible; rebuild from the retained dailies instead.

  • Can you compute how many visitors appeared in both of two weekly HyperLogLogs?
    Not reliably. Redis offers no intersection command, and computing |A|+|B|-|A union B| subtracts two large approximate numbers, so both absolute errors carry into a much smaller result. For a genuine intersection you need exact structures such as Sets or bitmaps over dense user IDs.
  • Does merging thirty daily keys into a monthly key make the estimate thirty times worse?
    No. The merge is a per-register maximum, so the result is identical to a key built by adding all the elements directly. The relative standard error stays around 0.81% for the merged cardinality, independent of how many sources were merged.

saying these in an interview costs you the question

  • Summing daily PFCOUNT values to get a weekly number, which double counts returning visitors
  • Claiming error accumulates or compounds with each PFMERGE
  • Believing an intersection command exists, or that inclusion-exclusion gives a usable intersection
  • Trying to subtract one day back out of a merged key
  • Ignoring that multi-key PF commands need one hash slot in cluster mode

context

open as a page

Redis exposes a HyperLogLog type through the PFADD and PFCOUNT commands. What problem does it solve, and what do you give up compared with putting the same items into a Redis Set and calling SCARD?

level: middleimportance: should knowfreq 45%

basics

~20 s

It counts unique items approximately in fixed memory. PFADD adds an element; PFCOUNT returns an estimated distinct count with about 0.81% standard error, using at most ~12 KB per key however many items you add. Unlike a Set it stores no members, so you cannot list them, test membership, or remove one.

open as a page

Redis documents a standard error of about 0.81% and a footprint of about 12 KB for a HyperLogLog. What do those two numbers actually mean for a counter tracking roughly 50 million uniques, and why is a freshly created key far smaller than 12 KB?

level: seniorimportance: should knowfreq 32%

basics

~20 s

0.81% is a standard deviation, not a bound: at 50 million uniques a typical estimate is off by ~400k, and a couple of percent happens. 12 KB is the dense form of 16384 six-bit registers; small keys use a compact sparse encoding and convert to dense once they exceed hll-sparse-max-bytes, irreversibly.

open as a page

Product wants unique-user counts sliced by page, country and hour, and also wants to answer 'has user X visited this page?' and 'how many users saw both page A and page B?'. How do you decide which of these Redis HyperLogLog can serve, and what do you use for the rest?

level: principalimportance: should knowfreq 24%

basics

~20 s

HyperLogLog serves only the sliced counts, via one key per slice plus PFMERGE rollups. Membership and intersection are impossible with it: no per-element test, no PFINTERSECT, and inclusion-exclusion is numerically useless. Serve those with Sets, bitmaps over dense user IDs, or an offline store, and budget ~12 KB per dense slice key.

open as a page