A Cloudflare Vectorize docs search filters on product, yet pages from the original import never match although they carry product metadata; why, and how do you fix it?
answer
- stored is not filterable
- a metadata index on the property
- created after the bulk load
- re-upsert, not insert
basics
~20 sVectorize filters only through metadata indexes, and a metadata index covers only vectors written after it was created. The product index was added after the import, so re-upsert the older vectors, and create metadata indexes before loading next time.
solid answer
~40 sMetadata stored on a vector is returnable but not filterable until the property has a metadata index from `wrangler vectorize create-metadata-index docs-search --property-name=product --type=string`. Vectors upserted before that index existed are not in it, and the filter runs before `topK`, so those pages are never candidates. I confirm with `list-metadata-index` and an unfiltered query showing the metadata, then re-upsert the older vectors: rerun the pipeline, or page IDs with `list-vectors`, fetch them with `getByIds()` and upsert unchanged. `insert()` won't do, since it skips existing IDs. Other quiet misses: only the first 64 bytes of a string are indexed, and a dotted key means nesting.
code
bash · 9 lines# Which properties are filterable, and with which type?
npx wrangler vectorize list-metadata-index docs-search
# Index the property (do this before loading vectors on a new index)
npx wrangler vectorize create-metadata-index docs-search --property-name=product --type=string
# Page through every vector ID to re-upsert the ones written before the metadata index
npx wrangler vectorize list-vectors docs-search --count=1000
npx wrangler vectorize list-vectors docs-search --count=1000 --cursor="<nextCursor>"go deeper
Recall that filtering on a metadata field needs a metadata index on it, created with create-metadata-index.
Explain that the filter runs before topK, which types a metadata index accepts, and that string values are indexed only to 64 bytes.
Diagnose from the timeline: vectors written before the metadata index are missing from it. Fix with a re-upsert of those vectors and confirm with a filtered smoke query for an old page.
Treat filterable fields as part of an index's schema, decided before the first load and versioned with the import job, since each later addition costs a full re-upsert.
## The symptom A documentation site's search runs on Vectorize. Every vector was written with metadata such as `{ product: "workers", language: "en", url: "..." }`, and the search Worker filters with `query(vector, { topK: 10, filter: { product: "workers" } })`. Pages added this month come back; pages loaded in the original bulk import never do, even for a phrase copied from their text. Removing the filter brings them back, with the right `product` in their metadata. ## Why it happens Vectorize filters on metadata only through **metadata indexes**. Storing a field in a vector's metadata (up to 10 KiB per vector) makes it returnable; it does not make it filterable. Filtering needs a metadata index on that property, created with `wrangler vectorize create-metadata-index`. The rule that explains the symptom: **vectors written before a metadata index existed are not in it**. A metadata index covers vectors inserted or upserted after it was created. The documentation says to create metadata indexes right after the index and before loading vectors. So the timeline was: 1. the index was created and the bulk import loaded every page; 2. someone added filtering later and ran `create-metadata-index` for `product`; 3. only vectors written since then are in that metadata index. The filter is also applied **before** `topK`: Vectorize takes the 10 matches from the filtered set. The old vectors are therefore not ranked low; they are never candidates, which is why even an exact phrase misses. ## How to confirm it - `wrangler vectorize list-metadata-index docs-search` shows which properties are indexed, and with which type. - An unfiltered query with `returnMetadata: "all"` shows that a missing page does carry `product`. - Compare when the import ran with when the metadata index was created; the pages that miss are those written before. - Upsert one missing page again by hand and repeat the filtered query once the mutation is applied; if that page now matches, the diagnosis is confirmed before the full backfill starts. ## The fix 1. Make sure the metadata index exists with the right type: `wrangler vectorize create-metadata-index docs-search --property-name=product --type=string`. Types are `string`, `number` and `boolean`. 2. **Re-upsert every vector written before it.** Rerunning the indexing pipeline with `upsert()` is simplest. Without the source pages, page through the IDs with `wrangler vectorize list-vectors docs-search --count=1000 --cursor=...`, fetch values and metadata with `getByIds()` (or `wrangler vectorize get-vectors`), and upsert them unchanged. `insert()` does not work here: it skips IDs that already exist. 3. Wait for the mutations to apply, since writes are asynchronous, then re-run the failing searches. ## Other ways a filter quietly misses | Cause | What to check | |---|---| | Property never indexed | `list-metadata-index` | | Index created after the vectors | re-upsert the older vectors | | String compared beyond 64 bytes | only the first 64 bytes of a string are indexed | | Dotted key | `{ "a.b": 1 }` looks for nested `{ a: { b: 1 } }` | | More than 10 properties wanted | each index allows 10 metadata indexes | | Metadata dropped on re-index | an upsert without metadata replaces it with none | A few filter rules sit alongside: operators are `$eq`, `$ne`, `$in`, `$nin`, `$lt`, `$lte`, `$gt` and `$gte`; several keys combine as an implicit AND; and the filter's compact JSON must be under 2,048 bytes. A **namespace**, set on the vector at write time, filters with no metadata index at all, which is one reason to use it for a partition chosen up front. ## Keeping it from happening again - Create the index and its metadata indexes in one setup script, before the first load. - When a new filterable field is added later, make the re-upsert part of the same change. - Add a smoke check after each import: a filtered query for a known old page must return it. Why filtering by metadata beats similarity alone, and how pre- and post-filtering trade recall, are covered with filtered retrieval; this question stays with Vectorize's metadata index rule.
- How do you re-upsert old vectors when the original pages are no longer at hand?Page through the IDs with `wrangler vectorize list-vectors docs-search --count=1000`, following `nextCursor` until it is null; the listing is a snapshot taken at the first request. Fetch each batch's values and metadata with `getByIds()` on the binding or `wrangler vectorize get-vectors`, and `upsert()` them unchanged in large batches. `insert()` would skip every ID because they already exist.
- Why can a filter on a long string property match the wrong vectors?A string metadata index holds only the first 64 bytes of each value, cut on a UTF-8 character boundary. Two titles that share their first 64 bytes look identical to the filter. Filter on a short, stable key such as a slug or a category instead, and keep the long text as unindexed metadata for display.
- Assumes every stored metadata field is filterable
- Expects a new metadata index to cover vectors already in the index
- Re-runs insert() to backfill a metadata index
- Thinks the filter only reorders the top matches
- Blames the embedding model when only filtered queries miss