skip to content

Scale-Out and Replicas

Split an ANN index across machines and every query visits every shard unless routing narrows it, and splitting a shard usually means rebuilding its index. Interviewers probe both costs.

on this pageshow

questions

6

A vector store at its memory limit or its query-rate limit: when do you add shards, when replicas, and what does a replica cost?

level: juniorimportance: must knowfreq 55%

answer

  1. two different ceilings
  2. slice of the data versus whole copy
  3. every query still visits every shard
  4. replica = vectors plus index again

basics

~20 s

Add shards when the index no longer fits: each holds a slice of the vectors. Add replicas when queries outrun one copy: each replica is a full copy of a shard's vectors and index, multiplying memory and repeating every write.

solid answer

~50 s

A **shard** holds a subset of the vectors and builds its own approximate nearest-neighbour index over that subset, so the shard count sets how much data the cluster can hold. Under random placement a query still searches every shard, so more shards add capacity but not query throughput: each one adds another search to every query. A **replica** is a whole copy of a shard that answers queries on its own, so read throughput and availability rise with the replica count. The price is a full copy of the shard's vectors and its graph or list structure in memory on each replica, every write indexed again on each copy, and a build or file copy before a new replica can serve. So a memory ceiling calls for shards (or a smaller index), a queries-per-second ceiling calls for replicas.

go deeper

for a junior

Recall the two definitions: a shard is a slice of the vectors with its own index, a replica is a full copy of a shard. Shards add capacity, replicas add throughput and availability.

for a middle

Explain why random placement makes every query visit every shard, so more shards cost more work per query, and list what a replica costs: memory, repeated writes, time to build or copy.

for a senior

Size it: vectors times dimensions times bytes plus index links per copy, times the replica count, against node memory and measured per-copy throughput. Say which ceiling you hit first and why.

for a principal

Frame replica count as a cost-for-availability trade the team sets per tier of traffic, and push back on adding shards as a throughput fix before placement that lets queries skip shards is considered.

## Two different ceilings A vector store that serves an **approximate nearest-neighbour (ANN) index** runs into one of two limits first: - **Memory (or disk) capacity**: the vectors plus the index structure no longer fit on the nodes you have. - **Query throughput**: the copy you have cannot answer queries as fast as they arrive, so latency climbs and requests queue. The two limits have different fixes, and mixing them up is the classic mistake. **Shards** address capacity. **Replicas** address throughput and availability. Neither one fixes the other limit. ## What a shard is A **shard** is a slice of the collection: a subset of the vectors, stored on one node, with its **own** ANN index (a graph, a set of inverted lists, whatever the store uses) built over just that subset. Doubling the shard count roughly halves what each node must hold. What sharding does *not* do, under random or hash placement, is reduce the work per query. The nearest neighbours of a query can sit on any shard, so the query is sent to all of them and their results are combined (the merging itself is covered with query fan-out over sharded indexes). Each extra shard therefore adds one more index search to every single query. A graph search costs roughly logarithmically in the size of the graph, so many small searches add up to more total work than one large search over the same vectors. Placements that let a query skip shards exist, but they are a separate design choice. ## What a replica is, and what it costs A **replica** is a complete copy of one shard, its vectors **and** its index, on another node. Any replica of a shard can answer a query for that shard, so a cluster with three replicas of everything can serve roughly three times the queries of one copy, and survives a node loss without going dark. The costs: 1. **Memory per copy.** The index is not shared. Every replica holds the full vectors plus the graph links or list structure for its shard. 2. **Write amplification.** Every insert, update and delete must be applied, and usually indexed, on each copy. Some stores build index segments once and ship them, which saves the indexing CPU but not the storage. 3. **Time to a new replica.** A new copy serves nothing until its index has been rebuilt or its built files copied over and caught up with the writes made meanwhile. ## The arithmetic in one scenario These figures are a scenario's, not any store's: - 40 million vectors at 768 dimensions in 32-bit floats: 40,000,000 x 768 x 4 bytes = **122.88 GB** of raw vectors. - A graph index with about 32 neighbour links per vector on its base layer, 4 bytes per link: 40,000,000 x 128 bytes = **5.12 GB** of links. - One full copy is therefore about **128 GB**. On nodes with 64 GB usable for the index, that needs at least two shards; four shards of about 32 GB each leaves headroom. - Two replicas of every shard: 8 shard copies, about **256 GB** in total, twice the memory of one copy. Now suppose one copy of the whole set serves 400 queries per second and traffic grows to 1,200. Adding shards does not help: each query still runs on every shard. Three replicas per shard roughly triples the throughput, at three times the memory of one copy (about 384 GB) and three times the write work. ## Choosing between them | Symptom | Add shards | Add replicas | |---|---|---| | Index no longer fits in memory | Yes: each node holds less | No: each copy is the full shard | | Queries per second too high | No: every query still hits every shard | Yes: more copies answer in parallel | | One node loss takes data offline | No | Yes: another copy keeps serving | | Write cost | Unchanged per vector | Multiplied by the replica count | A memory ceiling can also be met without new shards, by compressing the vectors or moving to a disk-resident index; fitting an index into memory is covered with vector database index choices. ## Common mistakes - Expecting more shards to raise throughput: under random placement they add work to every query. - Pricing a replica as "just the vectors": the index is duplicated too. - Adding replicas to fix a memory ceiling: each replica hits the same ceiling. - Assuming a new replica can serve as soon as it is created.

  • Why does adding shards not raise query throughput when shards are placed at random?
    Because a randomly placed shard can hold any query's nearest neighbours, so every query is sent to every shard. More shards means more index searches per query, not fewer. A graph search costs roughly logarithmically in graph size, so several small searches cost more in total than one large one. Throughput comes from replicas, or from a placement that lets queries skip shards.
  • Can a team fix a memory ceiling by adding replicas instead of shards?
    No. Each replica holds the whole shard, vectors and index, so every copy hits the same ceiling as the first. A memory ceiling is met with more shards, compressed vectors or a disk-resident index; replicas only add throughput and availability.

  • Adding shards raises query throughput the same way adding replicas does.
  • A replica is cheap because it stores only the vectors, not the index.
  • Adding replicas lowers the memory each node needs.
  • Shard count only affects storage, never the work done per query.
  • A new replica can serve queries the moment it is created.
open as a page

Why does a vector store with randomly placed shards send every query to every shard, and which placements let a query skip shards?

level: middleimportance: must knowfreq 50%

basics

~20 s

Random or hashed placement can put any vector on any shard, so a query's neighbours could be anywhere and every shard is searched. Placing by tenant key skips shards exactly; placing by vector cluster skips them approximately, risking recall.

open as a page

Why can a vector store usually move an index shard to another node by copying it, but not split one without rebuilding its index?

level: middleimportance: should knowfreq 40%

basics

~20 s

A moved shard keeps every vector, so its index stays valid and copies as files. A split strands graph links across the cut, so each half needs a new index, though segment-based stores can sometimes reassign whole segments instead.

open as a page

A vector store places vectors on shards by nearest cluster centroid and routes each query to the closest few shards: what recall does that risk, and how do you bound it?

level: seniorimportance: should knowfreq 30%

basics

~20 s

Neighbours near a cluster boundary, and queries far from every centroid, land on shards the router skipped, so recall drops without any error. Bound it by measuring against full fan-out, probing more shards, duplicating boundary vectors and re-clustering when data drifts.

open as a page

Adding a replica of a large graph-indexed vector shard: do you rebuild the index on the new node or ship the built index files?

level: seniorimportance: should knowfreq 30%

basics

~20 s

Shipping the built files is usually faster and gives an identical replica, but needs a consistent copy, a compatible index format and a replay of writes since the copy. Rebuilding often costs hours of CPU and yields a slightly different graph.

open as a page

A search feature has ten thousand tenants on one vector store near its memory limit: an index per tenant or pooled shards by tenant key, and what happens when one tenant outgrows its pool?

level: principalimportance: should knowfreq 40%

basics

~20 s

Neither pure shape usually holds: ten thousand tiny indexes each pay fixed overhead, and pooled shards let one tenant crowd its neighbours. Most teams pool small tenants by tenant key and promote large ones to dedicated shards past a threshold.

open as a page