skip to content

When is a vector index held in your web application's own memory enough, and what breaks once the app runs as several replicas?

level: juniorimportance: must knowfreq 55%

answer

  1. one process, one copy
  2. small, read-mostly corpora
  3. each replica builds its own copy
  4. a write lands on one replica
  5. restarts and scale-ups start empty

basics

~20 s

An in-process vector index is enough for a small, read-mostly corpus served by one process that can rebuild it at startup. With several replicas each holds its own copy, so a write reaches one copy, results diverge and every restart reloads.

solid answer

~50 s

An in-process index is a library-level index living in the application's own memory: no network hop and nothing extra to run. It is fine when the corpus is small, changes rarely, and can be rebuilt or loaded from a prebuilt file when the process starts. Run four replicas behind a load balancer and you have four independent indexes. A document added through one replica is searchable only there, so the same query answers differently depending on routing; every replica pays the memory again; and each restart, deploy or autoscaled instance must rebuild or reload before it serves a complete corpus. Durability is nobody's job either: anything written only to the index since the last load dies with the process. The way out is to move the index somewhere every replica shares, or to treat it as a read-only artifact built offline and shipped with each deploy.

go deeper

for a junior

Know what an in-process index is and the two things that make it fine: a small corpus and one process. Be ready to say that each replica gets its own private copy.

for a middle

Walk through what happens when a write lands on one of four replicas, and why restarts and autoscaled instances start empty. Mention that memory is paid once per replica.

for a senior

Show you can tell a legitimate in-process design, such as a per-release read-only artifact, from a prototype that outgrew itself, and explain why broadcasting writes is a replication protocol in disguise.

for a principal

Frame the move out of process as choosing who owns durability and concurrency, and set the trigger for it: live user writes or a replica count that makes the memory bill matter.

## What "in-process" means An **in-process vector index** is an approximate nearest-neighbour index (or a plain list of vectors searched exhaustively) built by a library inside the application's own process and held in its memory. There is no separate server: a search is a function call, with no network hop, no connection pool and no extra system to deploy, monitor or patch. It is usually the first shape a prototype takes, because a retrieval feature can be demonstrated in an afternoon. What the shape does not have is just as important. Nothing outside the process knows the index exists. Nothing persists it unless the application writes it out. Nothing coordinates it with any other copy. ## When it is enough The shape is a legitimate production choice, not only a prototype's, when all of these hold: - **One serving process**, or several that never need to agree with each other within a release. - **A small corpus** that fits comfortably in each process's memory with room to spare. - **Read-mostly data** that changes on a known cadence (nightly, per release), not continuously from users. - **A cheap rebuild**: the vectors are kept somewhere durable, so rebuilding the index at startup takes seconds, not hours. - **Tolerance for staleness** between rebuilds. A product catalogue or a documentation set that changes with each release, rebuilt in a build job and loaded at startup, meets every point. ## What breaks with several replicas Scale the web tier to four replicas and the same code now creates four indexes that share nothing. The failures arrive in a predictable order: 1. **Writes diverge.** A user uploads a document; the request lands on replica two, which adds its vectors to its own index. The other three never hear about it, so the same query returns different results depending on which replica the load balancer picks. 2. **Memory multiplies.** Each replica holds a full copy, so the index's memory cost is paid four times, and the replica count is now limited by index size as well as by traffic. 3. **Cold starts get slow.** Every deploy, crash restart and autoscaled new instance starts empty and must rebuild or reload before it can answer correctly. During that window it either refuses traffic or serves an incomplete corpus. 4. **Durability belongs to nobody.** Anything written only into the in-memory index since the last load is lost when that process exits. 5. **There is no single answer to "what is in the index?"**, which makes every later bug report about a missing or stale result hard to reproduce. ## Workarounds that keep it in-process, and why they disappoint | Workaround | What it fixes | What it leaves broken | |---|---|---| | Rebuild from the source of truth on every start | Restarts and new replicas converge eventually | Startup time grows with the corpus; writes between rebuilds still diverge | | Ship a prebuilt index file with each release | Identical, fast-loading copies on every replica | No live writes at all between releases | | Broadcast each write to every replica | Replicas usually agree | A replica that is down misses writes; ordering differs; a new replica needs a full copy from somewhere | | Sticky routing per user | One user sees their own writes | Other users still miss them; a replica restart loses the user's view | The broadcast option deserves a warning: once it needs a durable write log, ordering and a catch-up path for late replicas, the team is building a replicated vector server without the engineering that goes into one. ## The way out The fix is to give the index **one owner that every replica talks to**. The candidate shapes are an embedded store whose files a single writer process owns, a vector server the team runs, a managed vector service, or a vector extension inside the database already in production. Each moves durability, concurrency and upgrades to a different owner, and that choice is a separate question from this one. The one exception is the prebuilt read-only artifact, which stays in-process on purpose and accepts that live writes wait for the next release. ## How to answer in an interview Lead with the conditions that make the in-process shape correct, then name the first failure at several replicas (divergent writes), and finish with the move: either freeze the index into a per-release artifact or move it to a shared owner. Strong answers name divergent writes as the first failure and observe that the memory bill scales with the replica count.

  • Could you broadcast every write to all replicas to keep their in-memory indexes in sync?
    You can, but it is a fragile replication scheme. A replica that is restarting misses writes, two writes can arrive in different orders on different replicas, and a new replica needs a complete copy from somewhere. Once you add a durable write log and a catch-up path to fix that, you are running your own vector server without the testing behind a real one.
  • When is shipping a prebuilt index file with each deploy the right answer?
    When the corpus changes on a release cadence and live writes are not needed. A build job creates the index from the kept vectors, the file is versioned with the release, and each replica loads it at startup, so every copy is identical. The costs are startup time that grows with index size and staleness until the next release.

Each replica is a clerk with a private notebook copied from the master list at the start of the shift. A note one clerk adds never reaches the others' notebooks, and a clerk who arrives mid-shift starts with a fresh copy that knows nothing of the day's additions.

  • Four replicas of the same code share one in-memory index automatically.
  • Sticky sessions fix divergence between replicas' in-memory indexes.
  • Anything added to an in-memory index survives a process restart.
  • In-process indexes are too slow for production, so they are only for prototypes.
  • Adding replicas adds search capacity without adding index memory.