How do you add full-text search to a LanceDB table and combine it with vector search?

## Why this exists on an embedded database The pitch is that vectors already live beside the rest of your columns in a Lance table, so keyword search should not require standing up a second engine and keeping two copies of the corpus in sync. `create_fts_index` builds an inverted index over one or more string columns of the same table, and the query builder gains a text mode. ## The three query modes `table.search(...)` behaves differently depending on what you pass and what you declare: - A vector, or `query_type="vector"` — nearest-neighbour search over the vector column. - A string with `query_type="fts"` — keyword retrieval against the full-text index. Matching is lexical: tokenized terms, with scoring that rewards rare terms and penalizes long documents. - A string with `query_type="hybrid"` — both legs run, and their results are fused. Everything downstream is unchanged: `.limit()`, `.where()`, `.select()` and the terminal methods work the same way, and hybrid results carry a fused relevance score rather than a raw distance. ## What hybrid needs before it works Hybrid mode has to produce a vector from your query string. That means either an embedding function is registered on the table's schema — so the client can embed the query with the same model used at ingest — or you supply the query vector yourself alongside the text. A hybrid query on a table with bring-your-own vectors and no registered embedding function has no way to build the dense leg, and that is the single most common first-attempt failure. The second requirement is a reranker: two ranked lists have to become one. ## Fusion and why rank-based fusion is the default The two legs produce numbers that are not comparable. A lexical score is unbounded and depends on corpus statistics; a cosine distance sits in a fixed range and is smaller-is-better. Adding or averaging them directly is meaningless, and normalizing them per query is fragile because the score distributions shift with the query. Reciprocal rank fusion avoids the problem by discarding the magnitudes entirely and combining positions instead: a document ranked highly by either leg scores well, and a document ranked decently by both wins over one that is first in a single leg. That robustness is why it is the default. LanceDB ships alternatives — a linear combination that does weight the two score sources, and a cross-encoder reranker that rescores the merged candidate set with a model that reads query and document together — selected via `.rerank(...)`. The cross-encoder is far more accurate and far slower, so it belongs on a small merged candidate list, not on hundreds of rows. The choice between fusion strategies as a retrieval technique is a broader retrieval-design question; what matters at the product level is knowing which knob LanceDB exposes and what the default does. ## The freshness trap A vector search on a partially indexed table still returns complete results, because the engine scans the unindexed fragments as well. Full-text search is less forgiving in practice: rows added after the FTS index was built are not matched until the index is updated. The symptom is brutal to debug — a document is plainly in the table, `where` finds it, and keyword search does not — and it is worse in hybrid mode, where the dense leg does find the row so results look merely mediocre rather than broken. The fix is maintenance: refresh the index after ingest, either by running the table's optimize step or by rebuilding the FTS index with replace enabled. Put it in the ingest pipeline rather than the runbook. ## When hybrid earns its cost Dense retrieval is strong on paraphrase and weak on exact tokens — part numbers, error codes, surnames, acronyms — because those carry little semantic signal and embedding models routinely place them near unrelated text. Keyword retrieval is the mirror image. Hybrid is worth its extra latency exactly where queries mix both, which is most real search over technical or catalogue content. For a corpus of prose answered by conceptual questions, the dense leg alone is often as good and half the cost. Decide by measuring both against labelled queries rather than by defaulting hybrid on. ## Operational notes Building the FTS index costs a pass over the text column and adds storage proportional to vocabulary and document count. Keep it scoped to the columns you actually search — indexing every string column inflates build time and footprint for no retrieval benefit.