What does meta_fields_to_embed do on a Haystack document embedder?

## What it does mechanically A Haystack document embedder does not necessarily embed `Document.content` alone. Before calling the model it assembles a string: `<meta value 1><separator><meta value 2><separator><content>` using exactly the meta keys listed in `meta_fields_to_embed`, in that order, joined with `embedding_separator` (default `"\n"`). Keys that are absent from a given document are skipped rather than erroring. The resulting vector is written to `Document.embedding`; `Document.content` and `Document.meta` are untouched, and the store receives the document with its original text. Most document embedders expose the pair — `SentenceTransformersDocumentEmbedder`, `OpenAIDocumentEmbedder`, `AzureOpenAIDocumentEmbedder`, `HuggingFaceAPIDocumentEmbedder`, `CohereDocumentEmbedder` among them. The related `prefix` and `suffix` arguments do a similar thing with a constant string rather than per-document metadata. ## Why it exists Chunking destroys context. A 200-word chunk lifted out of the middle of a manual may read "This is disabled by default and must be enabled per environment." — a perfectly good sentence that no query can find, because it never names the feature. The information a searcher would use lives in the section heading, which the splitter left behind on the parent document or in a metadata key. Embedding the title alongside the chunk restores exactly that. It is the smallest possible version of the context-enrichment family: no extra LLM calls, no rewritten text, one constructor argument. ## Why it is not free **Token cost.** Every document carries the extra text, on every re-index. With an API embedder that is a real line item, and with a local model it is real latency. **Truncation.** The model's input window applies to the *assembled* string. Prepend a long meta value to a chunk already near the limit and the tail of the actual content is silently cut. Budget the meta text as part of the chunk size. **Signal dilution.** The vector is a single point representing everything you fed it. Adding a long, generic title to a short, specific chunk shifts the vector toward the title, so chunks from the same section start looking alike and retrieval loses the ability to distinguish them. Embedding a UUID, a file path, a timestamp or any high-cardinality identifier is pure noise — it cannot help matching and it costs tokens. **Asymmetry with the query.** Queries are embedded by a *text* embedder with no metadata to attach. So you are matching a metadata-enriched document vector against a plain query vector. That works when the metadata is the kind of thing users actually type — a product name, a document title, a section heading — and works badly when it is not. ## What it is not It is not a way to make metadata filterable. Filtering reads `Document.meta` through the filter syntax and is entirely independent of what went into the vector; a field can be filterable without being embedded, and vice versa. It is also not a way to store metadata — the meta dict is persisted regardless. And it does not append dimensions to the vector: the vector's dimensionality is fixed by the model. The distinction matters in an interview because the two mechanisms answer different questions. "Only search within the 2024 policy documents" is a filter. "Find the chunk about retries even though the chunk never says which service it is about" is `meta_fields_to_embed`. ## Choosing fields Good candidates are short, human-meaningful, and would plausibly appear in a query: document title, section heading, product or component name, sometimes the document type. Bad candidates are ids, hashes, file paths, timestamps, ingestion bookkeeping and anything with thousands of distinct values. A useful discipline is to treat it as a retrieval-quality experiment rather than a default: index the same corpus with and without the field, run your evaluation set against both, and keep the version that wins. Because the setting changes the vectors, switching it later requires re-embedding the entire corpus — and, since content-hash ids are unaffected by embeddings, a re-run with `OVERWRITE` will correctly replace them in place. ## A word on symmetry If the enriched documents perform badly for short keyword-ish queries, that is usually the asymmetry biting. The remedies are to shorten the embedded metadata, to restrict it to one high-value field, or to move the requirement into hybrid retrieval instead — a keyword match on the title achieves much of the same effect without touching the vector space.