After FT.CREATE over millions of existing Redis hashes, FT.SEARCH still misses products: how do you work out why?
answer
- FT.INFO before the query
- is the scan still running
- errors the counter sees, gaps it cannot
- ten rows is not the total
basics
~20 sStart with FT.INFO. Check indexing and percent_indexed for an unfinished background scan, hash_indexing_failures for rejected documents, and an OOM status for a scan that stopped. Then check the prefix, key type and SKIPINITIALSCAN, which fail silently, and finally the query.
solid answer
~50 sI read `FT.INFO` before touching the query. If `indexing` is 1 and `percent_indexed` is below 1, the background scan is unfinished and queries legitimately see part of the data, so the cure is to wait and to gate cutover on completion. If `hash_indexing_failures` is climbing, documents were rejected, and `last indexing error` with its key says why: a non-numeric `price` keeps the whole document out, not one field. A `background indexing status` of `OOM failure` means the scan hit the memory limit and stopped, and new documents are not indexed either. If all of that looks clean but `num_docs` is low, suspect what produces no error at all: a prefix that does not match, keys of the wrong type, or `SKIPINITIALSCAN`. Only then do I look at the query itself: the default 10-row page, a tag query without braces, or a timeout returning partial results.
code
plaintext · 12 lines127.0.0.1:6379> FT.INFO idx:products
...
num_docs 1840213
indexing 1
percent_indexed 0.4217
hash_indexing_failures 3
...
# fields excerpted and reformatted, values illustrative
127.0.0.1:6379> TYPE product:7
string
127.0.0.1:6379> FT.PROFILE idx:products SEARCH QUERY "@category:{laptops}"go deeper
Know that FT.INFO exists and shows num_docs and indexing, and remember that FT.SEARCH returns 10 rows by default while its first element is the total.
Explain why a new index over existing keys is partial for a while, and how a value that fails conversion keeps its whole document out of the index.
Work the diagnosis in order: scan state, rejection counters, the OOM status, then the silent causes, and only then the query. Say which ones leave no trace in the failure counter.
Make index cutover a checked deployment step with an alias, a completeness check and a rollback, so that a partial index cannot reach users however large the dataset gets.
## Split the problem first The symptom is that a product exists as a hash and a search does not return it. There are only two places that can be true: the document is **not in the index**, or the document is in the index and **the query does not reach it**. `FT.INFO` tells you which, so read it before rewriting any query. A direct check helps too: search for one known product by a word from its title, and compare the result against `HGETALL` of its key. ## Step 1: is the background scan finished? Existing keys are indexed by a background scan that starts after `FT.CREATE` returns. While it runs, the index answers queries from whatever has been indexed so far, so the reply is simply incomplete. - `indexing` is 1 while the index is being generated. - `percent_indexed` shows progress, where 1 means 100%. - `num_docs` grows as the scan advances. The scan yields to other commands: `search-bg-index-sleep-gap` sets how many iterations run before a brief sleep, 100 by default. On a busy server with millions of keys, this can take a long time, and that is working as designed. The fix is procedural: load large datasets before creating the index, and move the alias with `FT.ALIASUPDATE` only once `indexing` is 0. ## Step 2: were documents rejected? `hash_indexing_failures` counts documents that failed to index. A failure is not limited to one field. A hash whose `price` is `n/a` under a `NUMERIC` field is **not indexed at all**, so its title will not be found either. The same goes for an out-of-range `GEO` value or a vector blob of the wrong size. An update counts too: when a previously good document is rewritten with a bad value, the engine deletes it from the index rather than keep a stale entry, so one bad feed run can make known products vanish. `FT.INFO` also carries `last indexing error` and `last indexing error key`, globally and per field, which usually identify the bad data at once. Numeric failures mention `Invalid numeric value`. Fixing the value with `HSET` makes the document eligible again, because the write triggers indexing. ## Step 3: did the scan run out of memory? If `used_memory` passes `maxmemory` during the scan, the scan stops. In Redis Open Source it does not wait and retry by default. `FT.INFO` then reports `background indexing status` as `OOM failure`, and the consequence reaches beyond the existing keys. Writes also stop being indexed, each recording `Index background scan did not complete due to OOM. New documents will not be indexed.` Free memory or raise the limit, then drop the index with `FT.DROPINDEX`, without `DD`, and recreate it. ## Step 4: what produces no error at all? These cases leave the failure counter at zero, which is what makes them slow to find: | Cause | Why it stays silent | How to see it | |---|---|---| | Prefix mismatch | `product:` does not match `products:1`, so the key is never considered | Compare `prefixes` in `index_definition` with real keys | | Wrong key type | A string at `product:7` is skipped by an `ON HASH` index | `TYPE product:7` | | `SKIPINITIALSCAN` | Existing keys are not scanned; they enter only when written again | Check the creation script: `indexing` reads 0 and `num_docs` stays low | | Field outside the schema | Undeclared fields feed nothing | `attributes` in `FT.INFO` | ## Step 5: is the query the problem? Only now look at the query: - **The page.** `LIMIT` defaults to `0 10`. The first reply element is the total, so a document may be matching and simply not be on the page. - **Tag syntax.** A bare term does not search `TAG` fields, and punctuation inside braces needs escaping. - **Timeouts.** With the default `search-on-timeout` of `RETURN`, a query over its 500 ms default returns what it has so far. - **Parsing.** `FT.EXPLAINCLI` shows how the query was understood, and `FT.PROFILE` shows where the time went. ## Prevent the repeat Make completion part of the deployment, so that a partial index never takes traffic: 1. Script `FT.CREATE` under a versioned name, with the alias still on the old index. 2. Poll `FT.INFO` until `indexing` is 0. 3. Compare `num_docs` with the expected key count, and investigate any gap. 4. Check `hash_indexing_failures` and `background indexing status`. 5. Run `FT.ALIASUPDATE`, and keep the old index until the new one has proved itself.
- Why would you create a Redis index with SKIPINITIALSCAN on purpose, given how it hides documents?Because the keys that matter are all future ones, or because a scan would be wasted effort. `TEMPORARY` indexes are the case the docs name: they are lightweight, and you can create thousands. On a large existing dataset it skips a costly scan, but any key not written since stays invisible to queries, so the choice has to be recorded.
- How do you make sure a Redis index rebuild never serves partial results to users?Query through an alias. Create the new index, leave the alias on the old one, and poll `FT.INFO` until `indexing` is 0, `num_docs` matches the expected count and `hash_indexing_failures` looks acceptable. Then run `FT.ALIASUPDATE`. Rolling back means pointing the alias at the old index again, so keep it until the new one has proved itself.
- A query during the background scan returns an error or blocks until done.
- A non-numeric value in a NUMERIC field only loses that one field.
- If hash_indexing_failures is zero, every matching key is indexed.
- A string key under the prefix counts as an indexing failure.
- A scan that hits maxmemory only affects the keys it had not reached.
- FT.SEARCH returning 10 rows means only 10 documents match.