When do you shard a FAISS index across GPUs instead of replicating it?

## Two different scaling axes Multi-GPU FAISS solves two problems that people routinely conflate. **Replication** scales *throughput*: N copies of the same index, queries distributed among them, roughly N times the queries per second. **Sharding** scales *capacity*: one index split into N disjoint pieces, so you can hold N times as many vectors as one device fits. Choosing the wrong one produces a system that is fast and too small, or large and no faster. ## The API ```python co = faiss.GpuMultipleClonerOptions() co.shard = True # False (default) = replicate co.useFloat16 = True index = faiss.index_cpu_to_all_gpus(cpu_index, co=co, ngpu=4) ``` The result behaves like an ordinary FAISS index — `search` returns a merged global top-k either way — so the difference is invisible in the calling code and entirely visible in the resource profile. Underneath, the replicated form is an index that fans queries out to identical copies, and the sharded form is one that broadcasts to slices and merges their results. ## What replication costs and buys Each GPU holds the complete index, so device memory usage is N times the index size and the maximum dataset is still whatever one device holds. In exchange, a batch of queries is split N ways and each device works on its slice independently — near-linear throughput scaling, and no cross-device coordination. One nuance matters operationally: a replicated setup only scales throughput if queries arrive in batches large enough to divide meaningfully. Sending one query at a time to a four-way replicated index leaves three GPUs idle, so a batching layer in front is part of the design, not an optimisation. ## What sharding costs and buys Each GPU holds roughly `n/N` vectors, so the aggregate dataset scales with device count. Every query, however, must visit every shard — each computes its own top-k and the results are merged. Consequences: - **Throughput does not scale.** All devices are busy on every query rather than working on different queries. - **Latency is set by the slowest shard.** Uneven shard sizes or a device shared with another workload drags the whole query. - **A merge step is added** to combine per-shard results into the final top-k. Cheap relative to the search, but real. ## Combining them At scale the two compose: shard until the index fits across a group of devices, then replicate that group as many times as your QPS demands. A 400 GB index at 5,000 QPS is not a choice between shapes — it is four-way sharding for capacity, replicated three times for throughput, twelve devices in total. Framing the answer this way is what distinguishes an engineer who has operated multi-GPU search from one who has read the flag list. ## Correctness and recall notes Sharding an IVF index means each shard has its own inverted lists over its own slice of the data, so the effective number of lists probed per query is per shard. Recall is not automatically identical to the unsharded index at the same probe setting, and a recall measurement should be re-run against the sharded configuration rather than inherited from a single-device benchmark. Ids also need care: the merged results must return globally meaningful ids, so the ids you add must be unique across the whole dataset, not per shard. ## Operational shape Replicas fail independently — losing one costs you throughput, and the remaining copies still answer every query correctly. Shards do not: losing one shard silently removes a slice of the corpus from every result. That asymmetry is the strongest operational argument for preferring replication whenever the index genuinely fits, and for treating a sharded deployment as needing redundancy of its own. ## The interview answer in one line Shard for memory, replicate for speed; check which one you are actually short of before reaching for the flag, and remember that a lost shard degrades correctness while a lost replica only degrades capacity.