Search and Indexing

## The query builder A LanceDB query is built, then executed. `table.search(query_vector)` hands back a query-builder object that has recorded only the query vector so far. Every method you chain onto it — the metric, the limit, a filter, a column projection, index-search knobs — mutates that plan and returns the builder again. No I/O happens until you call a terminal method that materializes rows. This matters for two reasons. First, mistakes in chaining are silent: a builder that is never terminated simply does nothing. Second, because execution is deferred, LanceDB can push the projection and the filter down into the columnar scan instead of reading whole rows and discarding them afterwards. ## The pieces you chain - `.limit(k)` — how many neighbours to return. This is the k of the nearest-neighbour search, not a page size applied afterwards. Set it explicitly; the builder has a small default that is easy to trip over when you assume you got everything. - `.metric("l2" | "cosine" | "dot")` — which distance function to score with. It must agree with the metric the vector index was built under; scoring with one metric against an index trained for another gives you results that are quietly wrong rather than an error. - `.select(["id", "text"])` — restrict the columns read. Lance stores data column-by-column, so skipping a large text or blob column genuinely reduces the bytes pulled off disk or out of object storage. - `.where("price < 100")` — a SQL-style predicate string over the table's scalar columns. ## Terminal methods `.to_pandas()` is the usual one in notebooks and returns a pandas DataFrame. `.to_arrow()` returns an Arrow table and is the cheapest, because Lance is Arrow-native and no conversion is needed. `.to_list()` returns plain Python dictionaries, which is what most application code wants. `.to_pydantic(Model)` maps rows onto a Pydantic model when the table was created from one. ## The _distance column Whatever shape you ask for, the results carry an extra `_distance` field alongside the table's own columns. It is a **distance**: lower is nearer, and rows come back sorted ascending. Under `cosine` this is cosine distance, so a perfect match tends toward 0, not toward 1 — a frequent source of confusion when someone wires the number straight into a UI labelled "similarity" or applies a threshold with the comparison inverted. If you need a similarity for display, derive it yourself from the distance. The values are also not comparable across metrics or across queries with different vector norms. Use them to order results and, if you must, to apply a cutoff you have calibrated empirically on your own data — never as an absolute quality score. ## Exactness On a table with no vector index, the returned neighbours are exact, because the engine scans every row. Once an approximate index exists, the same call returns approximate neighbours: the API is identical and nothing in the result signals which mode ran. That is deliberate — it lets you develop against exact results and add the index later — but it means correctness testing has to be explicit, comparing indexed results against a flat-scan baseline for a sample of queries. ## Practical shape of a call A typical production query names the metric, sets a limit, projects only the columns the caller needs, and terminates with `.to_list()` so no DataFrame machinery is dragged into a request path. Notebooks use `.to_pandas()` because inspecting `_distance` next to the source text is the fastest way to sanity-check that embeddings and metric agree.