A fraud feature counts events per card with Redis CMS.INCRBY, and cards with few events report counts in the hundreds: why, and how should the sketch be sized and used?
answer
- error is a share of the total
- threshold grows with the stream
- width from error, depth from probability
- rotate per window, confirm before acting
basics
~20 sCount-min estimates over-count through collisions, by up to about error times the sketch's total count, so low counts are noise. Set error from the smallest count you must trust, rotate sketches per time window, and confirm against exact data before acting.
solid answer
~40 s`CMS.INITBYPROB key error probability` bounds the over-count as a share of the whole sketch: results below `error * total_count` should be ignored. With error `0.001` and a million events in the window, that threshold is 1,000, so a card with five events can read in the hundreds. Size from the rule: to trust a count of about 2 in a million events you need error `0.000002`, which gives width 1,000,000 and, at probability `0.001`, depth 10, or 40 MB per sketch. At that point compare it with an exact `HINCRBY` hash. Use one key per window, since the count only rises and `CMS.INCRBY` rejects negatives. Because the error is upward, a busy-looking card may be innocent, so treat the sketch as a first gate and confirm against exact data before blocking.
code
plaintext · 15 lines> CMS.INITBYPROB fraud:declines:2026110310 0.000002 0.001
OK
> CMS.INFO fraud:declines:2026110310
1) width
2) (integer) 1000000
3) depth
4) (integer) 10
5) count
6) (integer) 0
> CMS.INCRBY fraud:declines:2026110310 card-42 1
1) (integer) 1
> CMS.INCRBY fraud:declines:2026110310 card-42 -1
(error) CMS: Number cannot be negative
> EXPIRE fraud:declines:2026110310 7200
(integer) 1go deeper
Recall that a count-min sketch estimates how often a named item occurred, that CMS.INCRBY adds and CMS.QUERY reads, and that estimates can come out too high.
Explain how error sets width and probability sets depth, compute the threshold from CMS.INFO's count, and predict the error replies for a missing key and a negative increment.
Diagnose inflated low counts as a threshold problem, size the sketch from the business rule, rotate it per window, and insist on an exact check before an action reaches a customer.
Decide whether a sketch earns its place at all against an exact hash per window, and who keeps the threshold under review as traffic grows. A sketch that outgrows its rule fails without any error.
## What the two arguments set `CMS.INITBYPROB key error probability` creates a **count-min sketch**: a grid of counters that items hash into. The RedisBloom source derives the grid's size like this: - **Width** is `ceil(2 / error)`, counters per array. - **Depth** is `ceil(log10(probability) / log10(0.5))`, the number of arrays. - **Memory** is `width * depth * 4` bytes, since each counter is a 32-bit unsigned integer. The docs' example confirms this: `CMS.INITBYPROB bikes:profit 0.001 0.002` reports width 2000 and depth 9 in `CMS.INFO`, which is 72 KB. `CMS.INCRBY` adds to one counter in each array, and `CMS.QUERY` returns the minimum of them. ## Why a quiet card reads high Other items that hash to the same counters inflate them. The minimum reduces that inflation without removing it, so an estimate is never below the true count unless a counter saturates. The docs state the bound as **`threshold = error * total_count`**, where `total_count` is the `count` field in `CMS.INFO`. Results below that threshold should be ignored and often treated as zero. Two things follow: - **The threshold grows with traffic.** It is tied to everything in the sketch, not to the item you asked about. - **`probability` is about the tail.** It is the chance that a below-threshold item collides with above-threshold items in every array, and so returns a much higher count. With error `0.001` and 1 million events in the sketch, the threshold is 1,000. A rule such as more than five declines in an hour then sits far below the noise floor. ## Sizing from the rule 1. **Find the smallest count the decision must trust.** Say a count of about 2 has to stand out. 2. **Estimate the total per window.** Say 1 million events per hour. 3. **Divide.** Error is `2 / 1000000`, or `0.000002`. 4. **Compute the grid.** Width is 1,000,000. Probability `0.001` gives depth 10. That is 40 MB per sketch. 5. **Compare with exact counting.** If only 200,000 cards are active per hour, one `HINCRBY` hash at a few tens of bytes per field comes to roughly 10 MB, and it is exact. 6. **Repeat when traffic grows.** The total in step 2 is an assumption, and a sketch sized for last year's volume no longer supports the rule. The sketch wins when distinct items vastly outnumber what you can afford to store, and when the counts you care about are high relative to the total. The docs make the same point: a uniform stream is a poor fit, and a skewed one with clear heavy hitters works well. ## Using it correctly | Command behaviour | Consequence | |---|---| | `CMS.INCRBY` on a missing key replies `CMS: key does not exist` | Create each window's sketch before writing | | `CMS.INITBYPROB` on an existing key replies `CMS: key already exists` | Treat that error as fine in a create-if-needed step | | A negative increment replies `CMS: Number cannot be negative` | Events cannot be undone, so windows come from rotation | | An estimate that saturates replies `CMS: INCRBY overflow` | Rarely reached, but it is a real error path | | `CMS.MERGE` needs an initialized destination of equal width and depth | Keep one size across windows; in Redis Cluster the keys must share a hash slot | The usual pattern is one sketch per hour, as the docs do per day for sales. Put an `EXPIRE` on each key, and merge the recent hours if a longer view is needed. A merged sketch sums the totals as well, so its threshold is correspondingly higher. ## What the error means for the decision - **False alarms, not misses.** While the key exists, a card over the limit will read over the limit. A card under it may also read over it. - **Gate, then confirm.** Use the sketch to decide which cards deserve an exact lookup, then block on the exact figure. Blocking directly on the estimate puts the cost of each collision on an innocent cardholder. - **Read on write.** `CMS.INCRBY` replies with the updated estimate, so a rule can test it in the same round trip without a separate `CMS.QUERY`. - **Watch the live threshold.** Read `count` from `CMS.INFO` and compute `error * count`. When the rule's limit falls below that, the sketch has stopped supporting the rule, even though every command still succeeds.
- Why not simply use a much wider sketch and never think about the threshold again?Memory is width times depth times 4 bytes, so each tenfold cut in error costs ten times the memory, and the total keeps growing with traffic. A sketch sized once for today's volume drifts out of range quietly. Recompute the threshold from CMS.INFO, and alert when the rule's limit approaches it.
- What does CMS.QUERY return for a card that has never been seen?Usually 0, but any value is possible, because the card's counters may have been raised by other items. The sketch has no notion of whether an item was seen, only a minimum across shared counters. A non-zero reply below the threshold says nothing about that card.
- A count-min sketch can under-report a card's events because of collisions.
- The error argument is a fixed absolute margin that does not change with traffic.
- Lowering probability is the way to shrink the size of the over-count.
- CMS.INCRBY with a negative number can undo an event that was recorded in error.
- A low CMS.QUERY result is reliable because the sketch only ever errs on big items.
- CMS.INCRBY creates its key on first use, like BF.ADD does.