How does LlamaIndex's IngestionPipeline avoid re-embedding unchanged documents?

## The construct ``` from llama_index.core.ingestion import IngestionPipeline, DocstoreStrategy from llama_index.core.storage.docstore import SimpleDocumentStore pipeline = IngestionPipeline( transformations=[splitter, extractor, embed_model], docstore=SimpleDocumentStore(), vector_store=vector_store, docstore_strategy=DocstoreStrategy.UPSERTS, ) nodes = pipeline.run(documents=docs, num_workers=4) ``` `run()` applies each transformation in sequence — parsing Documents into nodes, running metadata extractors, embedding — and returns the resulting nodes. If a `vector_store` is attached, embedded nodes are written there as well. On a second run over a mostly unchanged corpus, two different things stop work from happening, and interviews reward being precise about which is which. ## Mechanism one: the cache The pipeline carries an `IngestionCache`. For each transformation it computes a key from the input nodes and the transformation itself; if that key is present, the stored output is returned instead of running the step. This is what avoids paying an embedding provider twice for text that has not changed, and it also skips LLM-backed metadata extractors, which are frequently the most expensive stage in the pipeline. Things to know about the cache in production: - It is keyed by **content**, not by document id. Two identical chunks from different files hit the same entry. - It grows. `pipeline.cache.clear()` exists precisely because an unbounded local cache on a long-lived ingestion box is a real operational problem. `IngestionCache` can be backed by a remote store (Redis, for example) so that several workers share it. - Changing a transformation's configuration changes the key — a new chunk size or a new embedding model correctly invalidates, rather than silently reusing stale vectors. ## Mechanism two: the docstore The cache saves recomputation; it does not stop the pipeline from inserting the *same document again* under a new node id. That is the docstore's job. When a docstore is attached, the pipeline looks up each incoming Document by `id_` and compares its stored hash: - **Unseen id** → process and insert. - **Known id, hash unchanged** → skip entirely. - **Known id, hash changed** → under `DocstoreStrategy.UPSERTS`, delete the old nodes for that document and re-process it. `DocstoreStrategy` options are `UPSERTS`, `DUPLICATES_ONLY` and `UPSERTS_AND_DELETE`. The last also removes documents that exist in the store but are absent from the batch — correct only when you pass the *complete* corpus each run; pass a delta batch with that strategy and you will delete everything you did not resend. Upserting into a vector store requires that store to support deletion by document reference. Not every backend does this equally well, and "my upserts leave orphan vectors behind" is usually a store-capability problem, not a pipeline bug. ## The failure that dominates Almost every "my pipeline re-embeds everything every night" report is the same root cause: **document ids are not stable**. `SimpleDirectoryReader` generates a fresh id per Document unless you pass `filename_as_id=True`; a custom reader does whatever you wrote. With churning ids, the docstore sees an entirely new corpus, `UPSERTS` degrades to insert-everything, and duplicates accumulate in the vector store. The content cache may still spare you the embedding calls — which is why the symptom is sometimes "duplicates but no extra cost", and sometimes "extra cost too" if the text also changed slightly. Second most common: state that does not survive the process. An in-memory `SimpleDocumentStore` and the default cache both vanish when the job exits. Use `pipeline.persist("./pipeline_storage")` and `pipeline.load("./pipeline_storage")`, or back the docstore and cache with a real database, if the pipeline runs as a scheduled job. ## Operational shape `run(num_workers=N)` parallelises across a process pool; there is an async `arun()` for I/O-bound transformations. Sizing here is about the embedding provider's rate limits more than about CPU. And a pipeline that both dedups and caches still cannot fix an upstream that hands it slightly-different text every run — normalising whitespace and rendering deterministically in the reader is what makes hashing work at all.