Product wants unique-user counts sliced by page, country and hour, and also wants to answer 'has user X visited this page?' and 'how many users saw both page A and page B?'. How do you decide which of these Redis HyperLogLog can serve, and what do you use for the rest?
answer
- classify: count distinct / membership / intersection
- HLL does count + union only
- slice count x 12 KB drives sizing
- membership -> bitmap over dense user IDs
- intersection -> BITOP AND, not inclusion-exclusion
basics
~20 sHyperLogLog serves only the sliced counts, via one key per slice plus PFMERGE rollups. Membership and intersection are impossible with it: no per-element test, no PFINTERSECT, and inclusion-exclusion is numerically useless. Serve those with Sets, bitmaps over dense user IDs, or an offline store, and budget ~12 KB per dense slice key.
solid answer
~50 sSplit the requirements by the operation each needs. Counting distinct users per slice is exactly HyperLogLog's job: key per page/country/hour, PFADD on ingest, PFMERGE up to day and month. Size from slice count, not traffic: dense keys are ~12 KB, so a million live slices is ~12 GB. Membership is out. There is no per-element test, and PFADD's return value is not one. If 'has user X visited' must be answered, that is a Set per slice, or a bitmap indexed by a dense internal user ID with SETBIT/GETBIT, which is far cheaper than a Set when IDs are dense. Intersection is out too. PFMERGE is union only, and deriving |A and B| by inclusion-exclusion subtracts two large approximate numbers so the error can exceed the answer. Serve overlap with bitmaps and BITOP AND plus BITCOUNT, with exact Sets and SINTERCARD on small slices, or offline in the warehouse.
go deeper
Recognise that HyperLogLog answers only 'how many distinct', and that membership and overlap need a different structure.
Map each requirement to a structure and justify it, naming Sets and bitmaps as the alternatives for membership and intersection.
Add capacity arithmetic across slice dimensions, TTL and rollup strategy, and the concrete cost of bitmap-based intersection.
Own the boundary: which metrics may be approximate at all, how that is communicated and governed, and the memory budget the dimensionality implies before the product fixes its slicing.
## Sort the questions by the operation they need Every analytics ask reduces to one of: count distinct, test membership, or combine sets. HyperLogLog implements count distinct and union. It implements neither of the others, and no cleverness fixes that, so the first architectural move is to classify each requirement rather than negotiate with the structure. - Unique users per page/country/hour: count distinct. HyperLogLog fits. - Unique users across a week or across all countries: union. PFMERGE fits. - Has user X visited page P: membership. Does not fit. - Users who saw both A and B: intersection. Does not fit. ## Sizing the part that does fit The cost driver is the number of live slice keys, not traffic volume, because per-key cost is capped. Pages times countries times hours multiplies fast: 5,000 pages x 50 countries x 24 hours is six million hourly keys per day. At the dense ceiling of about 12 KB that is roughly 72 GB, which is a cluster-sizing decision, not a detail. Mitigations in the order I would apply them: cut dimensionality (do you truly need page x country, or page and country separately?); shorten retention of the finest granularity with TTLs, keeping only rolled-up dailies; and lean on sparse encoding, since long-tail slices carry small cardinalities and stay far below 12 KB. If the long tail dominates, raising hll-sparse-max-bytes trades CPU on access for a large memory saving. ## Serving membership Two real options. A Set per slice is exact and supports SISMEMBER, but stores every member — acceptable only for a bounded number of slices. A bitmap is usually better: map users to a dense integer ID at signup, then SETBIT page:P:2026-08-14 <userId> 1 and GETBIT to test. A bitmap costs about one bit per user in the ID space, so ten million users is about 1.25 MB per dense slice — far bigger than a HyperLogLog, so keep bitmaps only for the slices that genuinely need membership. ## Serving intersection The inclusion-exclusion trick, |A and B| = |A| + |B| - |A union B|, is what candidates reach for, and being able to argue about it is the point. Each term carries roughly 0.81% relative error of its own magnitude. If A and B are each 50 million and the real overlap is 200,000, you subtract numbers near 100 million with absolute errors near 400,000 each to produce a result of 200,000. The error dwarfs the answer and can even come out negative. Say this explicitly; it is the difference between knowing the API and understanding the structure. The workable answers are bitmaps with BITOP AND then BITCOUNT (exact, memory-heavy, and the AND cost is proportional to bitmap length so keep it off the hot path), exact Sets with SINTERCARD where slices are small, or pushing overlap analysis to an offline system that already holds the event log. A hybrid is common: HyperLogLogs for the broad always-on dashboard, plus bitmaps maintained only for the specific pages a product team is currently analysing. ## The governance part Finally, decide where approximation is allowed at all. Anything invoiced, contractually reported, or feeding fraud decisions needs exact counting; an internal traffic dashboard does not. Write that boundary down and label approximate metrics visibly, because the failure mode is not technical — it is a finance team reconciling a HyperLogLog number against an exact one and finding a 1% gap nobody can explain.
- Why is inclusion-exclusion over two HyperLogLogs unacceptable for a small overlap?Because each term's absolute error scales with its own magnitude. Subtracting two 100-million-scale estimates, each carrying hundreds of thousands of absolute error, to obtain a result in the low hundreds of thousands leaves an error comparable to or larger than the answer. It can even produce a negative number.
- When would you choose a bitmap over a HyperLogLog for unique counts?When user IDs are dense integers and you also need membership or intersection, or when exactness matters and the population per slice is small relative to the ID space. A bitmap gives exact BITCOUNT cardinality, GETBIT membership and BITOP set algebra, at roughly one bit per possible ID per slice.
saying these in an interview costs you the question
- Proposing PFADD's return value as a membership test
- Proposing inclusion-exclusion for intersections without acknowledging the error blow-up
- Sizing by traffic volume instead of by number of live slice keys
- Assuming HyperLogLog keys always cost 12 KB, or always cost little
- Allowing an approximate counter to back a billed or audited number