How would you switch embedding models for a Chroma collection already serving production traffic?

## Why this is a rebuild The question is not really about Chroma's API — it is about the fact that stored vectors carry no way back to text. A vector is the output of one specific model; you cannot translate it into another model's space, so "switch the model" always means "recompute every vector". Chroma makes this concrete in two ways: the collection's dimensionality is fixed by its first write, and nothing in the store records which model produced which vector, so a partial swap yields a collection whose distances are meaningless with no error to tell you. That rules out the tempting shortcuts — passing the new embedding function to the existing collection, or re-adding only changed documents. The first only affects how new text is embedded; the second leaves a mixed index. ## Prerequisite: the source of truth is not Chroma The rebuild is only possible if you can replay the corpus. If Chroma holds the sole copy of your documents you are dependent on reading them back out and hoping the stored text is exactly what was embedded — plausible, but fragile, and it fails outright for anything where the stored document differed from the embedded chunk. The durable arrangement is that the vector store is a *derived* index: object storage, a relational store, or the upstream document pipeline holds canonical text, and Chroma can be dropped and rebuilt at any time. Establishing that is worth doing before you need it, and a model migration is usually the moment teams discover they have not. ## The plan **1. Provision a new collection.** Versioned name (`docs_v2`), the new embedding function attached, and a distance metric matching what the new model was trained for — do not assume the old collection's metric carries over, since models differ in whether cosine or dot product is their native scoring. Put the model identifier in the collection metadata. **2. Size the backfill before starting it.** Multiply corpus size by per-document embedding cost and by throughput. For a hosted embedding API this is a real invoice and a real rate limit; for local inference it is GPU hours. Anything large is a resumable batch job with deterministic ids (content hashes work well), checkpointing, and per-chunk retry — not a script you run in a terminal and hope survives. **3. Backfill.** Chunk the writes, reconcile `count()` against what your job emitted at each phase, and sample records to confirm text and vector correspond. Deterministic ids make a retried chunk harmless. **4. Evaluate before you commit.** This is the step most often skipped and the one that justifies the whole exercise. Keep a fixed set of representative queries with known-good expected documents, and score both collections on it — recall at k, or whatever your product actually cares about. A larger or newer model is not automatically better on your domain, and the dimension increase you are paying for in storage and latency has to show up as measured quality. If it does not, the honest outcome is to abandon the migration, and having run the comparison is what makes that decision defensible. **5. Cut over.** The application should already read its collection name from configuration, so the flip is a config change and a restart, not a code deploy. A shadow-read period — serving from the old collection while querying the new one and logging the differences — buys real confidence when the traffic justifies it. **6. Keep the old collection for a rollback window,** then `delete_collection` it. Two full copies of the corpus is the cost of a safe migration; on a single-node Chroma deployment, check that the disk can hold both before you start, because the new model may also be wider than the old. ## The tradeoffs a lead actually owns - **Cost versus measured gain.** Re-embedding is the dominant expense and it recurs every time you change models. That argues for changing models rarely and deliberately, and for pinning the model version rather than tracking a moving alias. - **Dimensionality has a running cost.** A wider model raises memory and index size for as long as it is deployed, not just during the migration. That is an ongoing bill against a one-off quality measurement. - **The embedding model is an interface.** Treat it like a schema version: pinned, recorded in collection metadata, changed only through this process. Teams that treat it as a tunable config value end up with mixed collections and no way to prove it. - **Freshness during the window.** If documents are still arriving while you backfill, either dual-write new documents into both collections or replay everything written after your backfill's start watermark before cutting over. Deciding which *before* you start avoids a gap nobody notices until a recent document is unfindable. ## What good looks like afterwards A collection whose metadata names the model that built it, an application that reads the collection name from config, a source of truth outside the vector store, and an evaluation set that can be re-run on demand. Those four things turn the next model change from a project into a routine.