How does document metadata change what LlamaIndex's node parser emits?

## Metadata is not free decoration In LlamaIndex, a `TextNode` carries a `metadata` dict alongside its text. It is tempting to treat it as bookkeeping — file name, page number, ingestion timestamp, source URL — but the parser and the retrieval stack treat it as **part of the node's content**. Understanding that is what separates a candidate who has run a real pipeline from one who has read a quickstart. ## Propagation: documents to nodes Node parsers expose `include_metadata` (default `True`). With it on, every node produced from a document inherits that document's metadata dict — one shared set of keys smeared across potentially hundreds of nodes. Turning it off gives you bare text nodes and, with it, the loss of source attribution and of metadata filtering at retrieval time. In practice you almost always leave it on and control exposure per key instead. ## The budget effect LlamaIndex's text splitters are metadata-aware. Before splitting a document they render its metadata into a string, count its tokens with the same tokenizer they use for the text, and subtract that from `chunk_size`. The remainder is the effective budget for real content. The consequences are concrete: - With `chunk_size=512` and 100 tokens of metadata, each node holds about 412 tokens of document text — you get roughly 25% more nodes than you expected, and a proportionally larger embedding bill. - If the metadata is close to the budget, LlamaIndex warns that very little room is left. - If the metadata is **larger** than `chunk_size`, the effective size goes non-positive and the splitter raises a `ValueError` instead of producing degenerate nodes. Ingesting documents with a giant summary or a full abstract stuffed into metadata at a small chunk size is the usual way to trip this. The fix is rarely to raise `chunk_size`; it is to stop putting large text in metadata, or to exclude the heavy keys from the embed and LLM views. ## Two exclusion lists, two audiences Every node has `excluded_embed_metadata_keys` and `excluded_llm_metadata_keys`. They exist because the two consumers want different things: - **The embedding model** benefits from metadata that adds retrieval signal — a document title, a section heading, a product name — because those tokens make the vector match queries phrased in those terms. It is actively harmed by high-cardinality noise: UUIDs, timestamps, file paths, checksums. Those tokens dilute the vector and can make unrelated chunks from the same file look similar to one another. - **The LLM** benefits from provenance it may need to cite (source, page, date) and is harmed by internal plumbing keys that invite the model to hallucinate about them. So the common configuration is: exclude ingestion plumbing from the embed view, keep title and section in it; exclude vector-store bookkeeping from the LLM view, keep source and page. ## Seeing what was actually embedded The single most useful debugging call here is `node.get_content(metadata_mode=MetadataMode.EMBED)`, with `MetadataMode` imported from `llama_index.core.schema`. The enum has `ALL`, `EMBED`, `LLM` and `NONE`. Calling it with `EMBED` reproduces byte-for-byte the string the embedding model received; with `LLM`, the string the prompt will contain. When retrieval quality is inexplicably bad, printing this for a handful of nodes very often reveals that half of every embedded string is a repeated file path, or that the actual answer text was crowded out. ## Setting exclusions early Exclusion lists live on the document as well as the node, and parsers propagate them along with the metadata, so the cheapest place to configure them is at ingestion, on the `Document` object, before any splitting happens. Setting them after the fact means walking the node list and re-embedding. ## Interview-grade summary Metadata propagates automatically, consumes the same token budget as text, and is visible to both the embedder and the LLM by default. Good pipelines therefore keep metadata small, put genuinely retrieval-useful strings in it deliberately, exclude noisy keys from the embed view, and verify the result by printing the embed-mode content rather than assuming.