skip to content

A Cloudflare Vectorize docs search filters on product, yet pages from the original import never match although they carry product metadata; why, and how do you fix it?

level: seniorimportance: should knowfreq 24%

answer

  1. stored is not filterable
  2. a metadata index on the property
  3. created after the bulk load
  4. re-upsert, not insert

basics

~20 s

Vectorize filters only through metadata indexes, and a metadata index covers only vectors written after it was created. The product index was added after the import, so re-upsert the older vectors, and create metadata indexes before loading next time.

solid answer

~40 s

Metadata stored on a vector is returnable but not filterable until the property has a metadata index from `wrangler vectorize create-metadata-index docs-search --property-name=product --type=string`. Vectors upserted before that index existed are not in it, and the filter runs before `topK`, so those pages are never candidates. I confirm with `list-metadata-index` and an unfiltered query showing the metadata, then re-upsert the older vectors: rerun the pipeline, or page IDs with `list-vectors`, fetch them with `getByIds()` and upsert unchanged. `insert()` won't do, since it skips existing IDs. Other quiet misses: only the first 64 bytes of a string are indexed, and a dotted key means nesting.

code

bash · 9 lines
bash
# Which properties are filterable, and with which type?
npx wrangler vectorize list-metadata-index docs-search

# Index the property (do this before loading vectors on a new index)
npx wrangler vectorize create-metadata-index docs-search --property-name=product --type=string

# Page through every vector ID to re-upsert the ones written before the metadata index
npx wrangler vectorize list-vectors docs-search --count=1000
npx wrangler vectorize list-vectors docs-search --count=1000 --cursor="<nextCursor>"

go deeper

for a junior

Recall that filtering on a metadata field needs a metadata index on it, created with create-metadata-index.

for a middle

Explain that the filter runs before topK, which types a metadata index accepts, and that string values are indexed only to 64 bytes.

for a senior

Diagnose from the timeline: vectors written before the metadata index are missing from it. Fix with a re-upsert of those vectors and confirm with a filtered smoke query for an old page.

for a principal

Treat filterable fields as part of an index's schema, decided before the first load and versioned with the import job, since each later addition costs a full re-upsert.

## The symptom A documentation site's search runs on Vectorize. Every vector was written with metadata such as `{ product: "workers", language: "en", url: "..." }`, and the search Worker filters with `query(vector, { topK: 10, filter: { product: "workers" } })`. Pages added this month come back; pages loaded in the original bulk import never do, even for a phrase copied from their text. Removing the filter brings them back, with the right `product` in their metadata. ## Why it happens Vectorize filters on metadata only through **metadata indexes**. Storing a field in a vector's metadata (up to 10 KiB per vector) makes it returnable; it does not make it filterable. Filtering needs a metadata index on that property, created with `wrangler vectorize create-metadata-index`. The rule that explains the symptom: **vectors written before a metadata index existed are not in it**. A metadata index covers vectors inserted or upserted after it was created. The documentation says to create metadata indexes right after the index and before loading vectors. So the timeline was: 1. the index was created and the bulk import loaded every page; 2. someone added filtering later and ran `create-metadata-index` for `product`; 3. only vectors written since then are in that metadata index. The filter is also applied **before** `topK`: Vectorize takes the 10 matches from the filtered set. The old vectors are therefore not ranked low; they are never candidates, which is why even an exact phrase misses. ## How to confirm it - `wrangler vectorize list-metadata-index docs-search` shows which properties are indexed, and with which type. - An unfiltered query with `returnMetadata: "all"` shows that a missing page does carry `product`. - Compare when the import ran with when the metadata index was created; the pages that miss are those written before. - Upsert one missing page again by hand and repeat the filtered query once the mutation is applied; if that page now matches, the diagnosis is confirmed before the full backfill starts. ## The fix 1. Make sure the metadata index exists with the right type: `wrangler vectorize create-metadata-index docs-search --property-name=product --type=string`. Types are `string`, `number` and `boolean`. 2. **Re-upsert every vector written before it.** Rerunning the indexing pipeline with `upsert()` is simplest. Without the source pages, page through the IDs with `wrangler vectorize list-vectors docs-search --count=1000 --cursor=...`, fetch values and metadata with `getByIds()` (or `wrangler vectorize get-vectors`), and upsert them unchanged. `insert()` does not work here: it skips IDs that already exist. 3. Wait for the mutations to apply, since writes are asynchronous, then re-run the failing searches. ## Other ways a filter quietly misses | Cause | What to check | |---|---| | Property never indexed | `list-metadata-index` | | Index created after the vectors | re-upsert the older vectors | | String compared beyond 64 bytes | only the first 64 bytes of a string are indexed | | Dotted key | `{ "a.b": 1 }` looks for nested `{ a: { b: 1 } }` | | More than 10 properties wanted | each index allows 10 metadata indexes | | Metadata dropped on re-index | an upsert without metadata replaces it with none | A few filter rules sit alongside: operators are `$eq`, `$ne`, `$in`, `$nin`, `$lt`, `$lte`, `$gt` and `$gte`; several keys combine as an implicit AND; and the filter's compact JSON must be under 2,048 bytes. A **namespace**, set on the vector at write time, filters with no metadata index at all, which is one reason to use it for a partition chosen up front. ## Keeping it from happening again - Create the index and its metadata indexes in one setup script, before the first load. - When a new filterable field is added later, make the re-upsert part of the same change. - Add a smoke check after each import: a filtered query for a known old page must return it. Why filtering by metadata beats similarity alone, and how pre- and post-filtering trade recall, are covered with filtered retrieval; this question stays with Vectorize's metadata index rule.

  • How do you re-upsert old vectors when the original pages are no longer at hand?
    Page through the IDs with `wrangler vectorize list-vectors docs-search --count=1000`, following `nextCursor` until it is null; the listing is a snapshot taken at the first request. Fetch each batch's values and metadata with `getByIds()` on the binding or `wrangler vectorize get-vectors`, and `upsert()` them unchanged in large batches. `insert()` would skip every ID because they already exist.
  • Why can a filter on a long string property match the wrong vectors?
    A string metadata index holds only the first 64 bytes of each value, cut on a UTF-8 character boundary. Two titles that share their first 64 bytes look identical to the filter. Filter on a short, stable key such as a slug or a category instead, and keep the long text as unindexed metadata for display.

  • Assumes every stored metadata field is filterable
  • Expects a new metadata index to cover vectors already in the index
  • Re-runs insert() to backfill a metadata index
  • Thinks the filter only reorders the top matches
  • Blames the embedding model when only filtered queries miss