How do you decide whether a FAISS deployment justifies GPUs at all?

## Ask three questions, in order ### 1. Does the index fit? GPU FAISS has no paging: everything the index needs must live in device memory simultaneously — the vectors or compressed codes, the id table, the coarse centroids, any precomputed tables, plus headroom for per-query workspace at your real batch size and concurrency. Work out that number before anything else, because it determines whether the conversation is about one GPU or about a sharded multi-device deployment with its own redundancy requirements and its own failure mode (a lost shard silently removes part of the corpus from every result). If a compressed index fits comfortably, GPU is on the table. If fitting requires aggressive quantization that costs recall you cannot afford, a CPU deployment with far more RAM per dollar may serve the same recall target more cheaply. ### 2. Is there enough parallel work? GPUs are throughput machines, and FAISS's GPU indexes are written for large query batches. Score your workload honestly: - **Bulk offline work** — corpus-against-corpus matching, deduplication, nightly re-scoring, evaluation sweeps — is the ideal fit. Millions of queries, no latency constraint, batches as large as memory allows. - **High-QPS retrieval that can be batched** works well, with a dynamic batching layer trading a few milliseconds of latency for a large throughput multiple. - **Low-QPS single-query serving with a tight p99** is the weak case. Per-call transfer and launch overhead dominate, and a well-tuned CPU inverted-file index answers in a millisecond or two with no batching machinery to build and operate. The honest principal answer often ends "not for serving" — and that is a stronger answer than reflexively provisioning GPUs. ### 3. Is building the bottleneck? This is the case people most often miss. Training a coarse quantizer over a large sample and adding hundreds of millions of vectors is heavy, parallel, latency-insensitive work — exactly what a GPU is for. If your pain is a nightly rebuild that takes eight hours, a GPU may compress it to under an hour without any change to how you serve. `faiss.index_gpu_to_cpu` then `faiss.write_index` gives you a device-independent artifact that CPU serving hosts load unchanged. Build on GPU, serve on CPU is a mature, common shape and it decouples the two decisions entirely. ## The cost comparison to actually run Compare **cost per query at your recall target**, not peak queries per second. That means: measure achievable QPS on the GPU configuration at the recall you require and at your production batch size, do the same for a tuned CPU configuration on comparably priced instances, and divide by hourly cost. GPU instances carry a large premium, so a 10x throughput advantage is not automatically a win. Include utilisation — a GPU idle two thirds of the day is paid for around the clock, whereas a CPU fleet scales down. ## The operational tax Beyond price, GPUs add real friction that belongs in the decision: - **Version coupling.** Driver, CUDA runtime and FAISS build must agree; upgrades become coordinated events rather than a package bump. - **Capacity scarcity.** GPU instance types are harder to obtain and to autoscale than general-purpose CPU instances, in most environments. - **Hard memory ceiling.** Exceeding device memory is a failure, not a slowdown — there is no swap. Growth planning must be explicit. - **Narrower index catalogue.** Only part of FAISS's index selection has a GPU implementation, so a GPU commitment constrains future index choices. - **A fallback path is mandatory.** Serving must degrade to CPU when GPU capacity is unavailable, which means keeping the CPU-serialised artifact and testing that path. ## What a principal-level answer sounds like It separates the axes — capacity, throughput, build time — instead of treating "use GPUs" as one decision. It quotes a cost-per-query method rather than a benchmark headline. It names the split deployment (build on GPU, serve on CPU) as a first-class option rather than a compromise. And it states the conditions under which it would revisit the choice — corpus growth past device memory, a change in traffic shape from batchable to strictly online, or a recall target that compression can no longer meet. There is no single right answer here, and an interviewer asking this is checking whether you can hold several constraints at once and still commit to one.