skip to content

A support bot's semantic cache in a Redis vector set serves wrong answers and keeps growing: how do you fix the similarity check and retire old entries?

level: seniorimportance: should knowfreq 20%

answer

  1. the score is not raw cosine
  2. EPSILON is half the cosine distance
  3. elements have no TTL
  4. eviction takes the whole key
  5. dropping a huge key blocks

basics

~20 s

VSIM scores rescale cosine from -1..1 to 0..1, so EPSILON 0.1 accepts cosine 0.8, not 0.9; tighten it. Elements have no TTL, so track age yourself and call VREM, because dropping or evicting the key removes everything at once.

solid answer

~40 s

The score is cosine rescaled to 0 through 1, computed as one minus half the distance, so `EPSILON 0.1` keeps scores of 0.9 and above, which is cosine 0.8. A team that meant cosine 0.9 must pass `EPSILON 0.05`, then calibrate on labelled question pairs and keep one key per embedding model. For growth, a vector set has no per-element TTL. Put a date in each element name, list old ones with `VRANGE`, and remove them with `VREM`, which reclaims memory at once; or give the answer key a TTL and treat a hit with no answer as a miss. Avoid rotating huge keys: on 8.10 a big vector set frees on the main thread even with `UNLINK`, and eviction removes the whole key at once.

code

plaintext · 12 lines
plaintext
# Lookup: score 0.95 or better, which is cosine 0.9 or better
> VSIM cache:v3 FP32 "<blob>" COUNT 1 EPSILON 0.05 WITHSCORES
1) "2026-11-03:q8841"
2) "0.9712"
> GET answer:2026-11-03:q8841
"<stored answer>"

# Miss path
> VADD cache:v3 FP32 "<blob>" 2026-11-03:q8842
(integer) 1
> SET answer:2026-11-03:q8842 "<model answer>" EX 604800
OK

go deeper

for a junior

Remember that a vector set returns a score between 0 and 1, and that its elements have no expiry of their own, unlike a key.

for a middle

Work the arithmetic: score is one minus half the distance, so a score limit and a cosine limit differ, and EPSILON applies to the rescaled side.

for a senior

Show the whole design: calibrated thresholds, a key per model version, answers kept out of attributes, batched VREM for retirement, and awareness that big-key drops and eviction are all-or-nothing.

for a principal

Ask what a false hit costs in support trust against a model call. That ratio sets the threshold, and it should be reviewed like any other product decision.

## The setup A **semantic cache** answers a new question from a stored answer when an earlier question means nearly the same thing, which saves a model call. The bot embeds each question, turning it into a vector, and asks a vector set for the nearest earlier question: 1. Embed the incoming question with the same model used for the stored ones. 2. Run `VSIM cache:v3 FP32 <blob> COUNT 1 EPSILON 0.05 WITHSCORES`. 3. On a result, fetch the stored answer from its own key, named after the element. 4. On no result, or if the answer key has gone, call the model. 5. Then `VADD` the question vector and `SET` the answer with a TTL. Keeping answers in separate string keys is deliberate. The docs advise keeping attributes minimal, since each one sits in memory on its graph node, and `VSIM` only needs a name to return. ## Why the bot serves wrong answers The usual culprit is a threshold written for a different scale. - **What `WITHSCORES` reports:** not raw cosine similarity. `vset.c` returns `1.0 - distance/2`, and the module README says the interval is rescaled from -1..1 to 0..1. The docs' own example shows it: vectors (1, 1) and (-1, 1) have cosine 0 and report a score of 0.5. - **What `EPSILON` limits:** `VSIM` stops at the first result whose `distance/2` exceeds it, so a result needs a score of at least `1 - EPSILON`. | `EPSILON` | Minimum score | Equivalent cosine | |---|---|---| | 0.2 | 0.8 | 0.6 | | 0.1 | 0.9 | 0.8 | | 0.05 | 0.95 | 0.9 | A team that carried over a "cosine 0.9" rule and wrote `EPSILON 0.1` is accepting cosine 0.8, which can be a different question. The docs' phrase "distance < 0.2" for `EPSILON 0.2` refers to this rescaled distance, so it is easy to misread. Three more causes to check: - **`BIN` quantization:** the score comes from normalized Hamming distance, so a threshold tuned on `Q8` will not carry over. - **A changed embedding model:** a new dimension fails with `ERR Vector dimension mismatch`, but a new model with the same dimension mixes incompatible vectors silently. Version the key, as in `cache:v3`. - **Scope:** an answer valid for one language or product may be near-identical in meaning for another. Use one key per scope when one value dominates, or `FILTER` on a flat attribute. Then calibrate. Label pairs of questions as same or different, read their scores, and pick the limit where false hits become acceptable. A wrong cached answer costs more than a model call, so bias towards misses. ## Why it keeps growing, and how to retire entries A vector set has **no per-element expiry**: `EXPIRE` works on the whole key and `HEXPIRE` belongs to hashes. Retirement is your job. | Approach | How it works | Watch out for | |---|---|---| | Dated names with `VRANGE` | Name elements `2026-11-03:q8842`, list old ones in lexicographical order, then `VREM` each | `VRANGE` needs 8.4 or later | | Sidecar sorted set | `ZADD` the time on write, read old names by score, then `VREM` | A second structure to keep in step | | Answer key TTL | Answer expires; a hit with no answer counts as a miss and triggers `VREM` | Entries nobody hits again linger | | Rotating keys | One key per day, searched together, each with `EXPIRE` | Dropping a big key blocks; see below | `VREM` does real deletion with relinking, not a tombstone, so memory returns immediately and the docs call it safe for long-running churn. It removes one element per call and waits for any background `VSIM` on that key, so pipeline small batches. ## Two behaviours that catch people out - **Eviction works on keys.** Under an `allkeys-*` `maxmemory-policy`, Redis can evict the entire vector set at once, never its oldest elements. A cold cache then hits the model all at once, so bound the size yourself. - **Dropping a big key is slow.** The docs warn that `DEL` or expiry of a very large set can spike latency. In 8.10 the type registers no free-effort callback, so `UNLINK` also frees it on the main thread. Rotating keys is fine at small sizes and a hazard at millions of elements.

  • Why not store the answer in the element's JSON attributes and read it with `WITHATTRIBS`?
    It works, since `WITHATTRIBS` has existed since 8.2, and it saves a round trip. But attributes live on the graph node in memory for every element, and the docs advise keeping them small. Long answers would outweigh the vectors, and they could not carry their own TTL. A separate string key with `EX` keeps the graph lean and gives expiry for free.
  • The team upgrades to a new embedding model with the same dimension. What do you do with the cache?
    Start a new key, such as `cache:v4`, and stop reading the old one. Vectors from different models are not comparable, and Redis cannot tell because the dimension matches, so a shared key would rank nonsense without any error. Let the cache refill, then retire the old key in batches with `VREM`, because dropping a big key at once can block.

  • WITHSCORES reports raw cosine, so a score of 0.8 means cosine 0.8.
  • EPSILON 0.1 means a cosine similarity of at least 0.9.
  • You can give one element a TTL with EXPIRE or HEXPIRE.
  • UNLINK makes dropping a huge vector set free of latency.
  • VREM only leaves a tombstone, so memory stays used until a rebuild.
  • LRU eviction removes the oldest elements from a vector set.

Assumes Redis 8.10; VRANGE arrived in 8.4, and the UNLINK behaviour was read from vset.c, module.c and lazyfree.c at 8.10.3.