skip to content

A Redis VSIM call with FILTER and COUNT 10 returns only three elements although more should match: how do you diagnose and fix it?

level: seniorimportance: should knowfreq 18%

answer

  1. effort cap or bad data
  2. COUNT times one hundred
  3. TRUTH removes the graph from the question
  4. a missing field fails everything
  5. VSETATTR never validates JSON

basics

~20 s

Either the search hit its effort cap, COUNT times 100 candidates by default, or the attributes do not match as you think. Compare against the same query with TRUTH, inspect attributes with VGETATTR, then raise FILTER-EF or split the set.

solid answer

~40 s

`FILTER` tests attributes on nodes as the graph walk reaches them, with no attribute index, and `FILTER-EF` caps that walk at `COUNT * 100` candidates by default, so rare matches can run out of budget. First rerun the query with `TRUTH`, an exact scan with the same filter. If it returns ten, the cap is the cause, so raise `FILTER-EF`, or split the collection into keys when one rare value dominates. If it also returns three, the data is the cause: check `VGETATTR`, since `VSETATTR` stores invalid JSON without complaint, and such elements never match. On 8.10 a selector for a missing field makes the whole expression false, even under `or`. Remember that a long read makes `VREM` and `VSETATTR` wait on the main thread, so cap this work.

code

plaintext · 8 lines
plaintext
> VSIM word_embeddings_bin ELE "pasta" FILTER ".len == 33"
1) "skinless_boneless_chicken_breasts"
2) "boneless_skinless_chicken_breasts"
3) "Boneless_skinless_chicken_breasts"
> VSIM word_embeddings_bin ELE "pasta" FILTER ".len == 33" FILTER-EF 10000
 1) "skinless_boneless_chicken_breasts"
 ...
10) "Peppermint_Mocha_Twist_Gingersnap"

go deeper

for a junior

Remember that FILTER tests JSON attributes attached to elements, and that an element with a missing or malformed attribute is left out with no error.

for a middle

Explain the effort cap: FILTER-EF defaults to COUNT times 100, and the search returns what it found when the cap is reached.

for a senior

Show the diagnosis: TRUTH separates the cap from the data, VGETATTR and VINFO check the attributes, and long reads can stall writers on the main thread.

for a principal

Decide per collection whether a dominant filter deserves separate keys, and set a limit on query effort, because silent short results will not show up in an error rate.

## The mechanism A vector set element can carry **attributes**, a JSON string set with `VSETATTR` or `VADD ... SETATTR`. `VSIM ... FILTER '<expression>'` returns only elements whose attributes satisfy the expression. There is **no index on attributes**. The search walks the HNSW graph, the layered links between nearby vectors, and evaluates the expression on nodes as it reaches them. Non-matching nodes still get explored, but they do not count as results. When matches are rare, the walk could cover the whole graph, so `VSIM` caps it: - **`FILTER-EF`** is the maximum number of candidates evaluated. - **The default** is `COUNT * 100`, so 1,000 for `COUNT 10`. - **At the cap**, the search returns what it has, without an error or a warning. The module README describes this as deliberate: on a rare filter, Redis prefers returning few elements to a large scan whose extra results may lie far from the query anyway. ## How to diagnose it Separate the two possible causes before changing anything. 1. **Rerun with `TRUTH`.** `VSIM ... FILTER '...' COUNT 10 TRUTH` scans all elements linearly, applying the same filter with no effort cap. It is O(N), so run it off-peak. 2. **Read the comparison.** Ten results with `TRUTH` and three without means the cap or the graph walk fell short. Three in both means only three elements really match. 3. **Check the attributes.** Use `VGETATTR` on an element you expected. `VINFO` reports `attributes-count` beside `size`, which shows how many elements have attributes at all. 4. **Check other limits.** An `EPSILON` in the same call trims results by distance, whatever the filter says. ## Why the data may not match The filter fails quietly on bad data, so a broken attribute looks like a rare value. - **No attributes:** an element without attributes never matches. - **Invalid JSON:** `VSETATTR` stores the string as given, with no validation. Such an element never matches. - **Missing field:** in 8.10's `expr.c`, a selector that finds nothing ends the evaluation with false. There is no short-circuit around it, so `.plan == "pro" or .tier == "gold"` fails on an element that has `tier` but no `plan`. - **Unsupported values:** JSON objects cannot be used in expressions, so `.movie` on a nested object never matches, and arrays work only if they hold just numbers and strings, for the `in` operator. - **Strings in numeric comparisons:** the docs say a string in `.rating > 8` excludes the element. In 8.10's `expr.c`, comparison operators parse the string as a number and use 0 if that fails. So `"9"` matches, and worse, `"high"` satisfies `.rating < 8`. Store real numbers. A nested path is different, because it fails loudly. Only top-level fields are reachable, and the parser reads `.movie.year` as two selectors in a row, so `VSIM` replies `ERR syntax error in FILTER expression`. That cannot be the cause of short results. Keep attributes flat anyway. ## How to fix it | Situation | What to do | |---|---| | Occasional rare filter, results still useful | Raise `FILTER-EF` for that query, for example to 10,000 | | A rare value that most queries filter on, such as a tenant ID | Use one key per value, so the graph holds only candidates | | Sparse results are acceptable | Keep the default, and let the application handle fewer results | | Far-away matches appear after raising the cap | Add `EPSILON` to drop results that are too distant | | Attribute problems | Fix the writer, and validate JSON before `VSETATTR` | The docs say `FILTER-EF 0` explores as many nodes as needed, but `VSIM` in 8.10's `vset.c` rejects any value of zero or below with `ERR invalid FILTER-EF`. Pass a large explicit number. A high value does not always cost much: when matches are common, early stop conditions end the walk long before the cap. ## The cost nobody budgets for `VSIM` runs on a background thread, up to 32 at once, so a slow search does not block the event loop by itself. It does hold a read lock on the key. `VREM`, `VSETATTR` and a `VADD` that updates an element wait from the main thread for background readers on that key to finish. So a large `FILTER-EF` or a `TRUTH` scan, while a writer removes one element, can stall every client of that Redis instance until the read completes. Bound the effort per query, and run `TRUTH` in a quiet window with writers paused.

  • Why not just raise `COUNT` to get more filtered results?
    Raising `COUNT` does raise the default cap, since `FILTER-EF` defaults to `COUNT * 100`, and it asks for more results too. But it mixes two meanings and returns more elements than the caller wants, if matches exist. Set `FILTER-EF` explicitly and leave `COUNT` as the number of results needed, which keeps the effort visible in the query.
  • A rare filter on a tenant ID is in almost every query. What do you change?
    Stop filtering and give each tenant its own vector set key. The search then walks a graph containing only that tenant's elements, so no candidates are wasted, and the graph is smaller and faster. The costs are per-key overhead and no search across tenants without a client-side merge of `VSIM ... WITHSCORES` results.

  • FILTER runs after the search, on the top COUNT results only.
  • An element with invalid JSON attributes makes VSIM return an error.
  • VSETATTR rejects a string that is not valid JSON.
  • FILTER uses an index on attributes, so rare values are found faster.
  • A very large FILTER-EF costs nothing because the search always stops early.
  • A slow VSIM on a background thread can never affect other clients.

Assumes Redis 8.10; the FILTER-EF check and the selector and string-number behaviour were read from vset.c and expr.c at 8.10.3, and differ from the docs.