What does Pinecone's selective metadata indexing buy you on a pod-based index?

## The mechanism When you create a pod-based Pinecone index you can pass a `metadata_config` object with an `indexed` list naming the metadata fields that should be filterable. Only the named fields get a filter index built for them. Everything else about the record is unchanged — the metadata is still stored, still travels back on queries that request it, and still costs storage. What it loses is the ability to appear in a filter expression. If you omit `metadata_config` entirely, all metadata fields are indexed. Passing an empty `indexed` list is the opposite extreme: nothing is filterable, which is a legitimate choice for an index used purely for display metadata. ## Why the knob exists: pod memory economics A pod has a fixed memory budget, and it holds both the vector index and the metadata filter indexes. Every additional indexed field takes a share of that budget, and the size of that share scales with **cardinality** — the number of distinct values the field takes. - `is_public` (2 distinct values) is nearly free. - `doc_type` (a dozen values) is cheap. - `tenant_id` (a few hundred values) is real but usually worth it. - `chunk_id`, unique per vector, is the pathological case: its index is as large as your record count and buys you nothing, because filtering by a unique id is what fetching by vector id is for. The practical effect of indexing a high-cardinality field is that fewer vectors fit per pod, so you scale to more pods sooner and pay for capacity that is holding an index nobody queries. Selective metadata indexing is how you avoid that. ## The tradeoff you are accepting The configuration is set when the index is created. You cannot decide six months later that `language` should have been filterable and flip a switch — you create a new index with the wider `indexed` list, re-upsert, and cut over. That makes this a decision worth taking deliberately rather than by default. The pragmatic middle ground: index the fields you filter on today plus the one or two you can clearly foresee, and leave genuinely display-only fields (titles, URLs, snippets, source paths) out. Those are exactly the fields that tend to be high-cardinality, so the memory win and the "never filtered anyway" property usually coincide. ## What happens when you filter an unindexed field This is the failure mode people hit: the field is visibly present in the returned metadata, so it looks like it should be filterable, but the filter does not behave as expected. Debugging it means going back to the index configuration rather than staring at the data. When someone reports "the filter isn't working but I can see the value right there in the response", check `metadata_config` first. ## Serverless is a different world Serverless indexes do not expose selective metadata indexing — you do not configure pods, so there is no pod memory budget for you to protect. That does not make the underlying discipline irrelevant: on serverless, a high-cardinality filter field still shapes query cost and latency, because a filter that matches a tiny, widely scattered slice of the index forces more of it to be read. The lever changes from "which fields are indexed" to "how the data is laid out and how selective the filters are". ## How to decide, field by field For each metadata field, ask three questions: 1. **Does any query filter on it?** If no, it does not need indexing — only storage. 2. **How many distinct values does it take?** Low cardinality is cheap; per-record uniqueness is a red flag. 3. **Is it a per-query constraint or a dataset partition?** Constraints that vary per query are what filters are for. A dimension that permanently splits your dataset into disjoint groups is usually better handled by the index layout than by paying for a filter index on every record. Write that decision down next to the schema. Because the configuration is immutable, the reasoning behind it is the thing future engineers most need and are least likely to reconstruct. ## Diagnosis in production Symptoms that point back here: pods filling up far faster than vector count alone predicts; a filter that silently fails on a field you can see in the response; a plan to add a new filterable dimension that turns out to require a full re-index. All three are cheaper to prevent at creation time than to fix later, which is why this comes up as a design-review question rather than a coding one.