Why is a FAISS GPU index barely faster than CPU when you search one query at a time?
answer
- fixed cost per call, tiny work
- wide machine, one lane used
- overhead dominates at nq = 1
- throughput metric, not latency metric
- queue briefly, dispatch together
basics
~20 sA single query leaves the GPU almost idle: fixed per-call costs — transferring the query, launching kernels, copying results back — dominate the tiny amount of work. FAISS GPU indexes are throughput engines; you get the speedup by searching thousands of queries in one call.
solid answer
~50 sFAISS's GPU kernels are written to saturate thousands of cores with a large query batch. Search one vector and you pay the full fixed cost of the call — copying the query to the device, launching the kernels, copying distances and ids back — for an amount of arithmetic that uses a fraction of the hardware, so wall-clock time is dominated by overhead rather than search. Passing a `(nq, d)` array with nq in the thousands amortises all of it, and that is where the order-of-magnitude speedups quoted for GPU FAISS come from. The practical consequence for serving: if your traffic is single online queries with a tight latency budget, a GPU may not beat a well-tuned CPU index, and the way to make it pay is a batching layer that accumulates arriving queries for a few milliseconds and dispatches them together — spending a little latency to buy a lot of throughput.
code
python · 12 linesimport numpy as np, time
queries = np.random.random((8192, 128)).astype('float32')
t0 = time.time()
for q in queries: # overhead-dominated
gpu_index.search(q.reshape(1, -1), 10)
per_query_loop = time.time() - t0
t0 = time.time()
D, I = gpu_index.search(queries, 10) # one batched call
batched = time.time() - t0go deeper
Remember that FAISS GPU search takes a matrix of queries and is measured in queries per second on a large batch — searching one at a time is not how the speedup is obtained.
Explain the mechanism: fixed transfer, launch and synchronisation costs per call versus how little parallel work one query offers, and how a large batch amortises all of it.
Turn it into a serving design — dynamic batching with a bounded wait, batch size chosen from a throughput sweep and constrained by device memory under concurrency, and the honest case for CPU serving.
Own the cost question: whether GPU serving beats CPU per query at your traffic shape at all, or whether the GPU belongs at build time while serving stays on cheaper hardware.
## Throughput hardware, throughput API A GPU is a wide machine: thousands of cores that only pay off when there is enough parallel work to fill them. FAISS's GPU indexes are written accordingly — `search` takes a matrix of queries, and the kernels are laid out to process many queries against many database vectors simultaneously. With one query, most of the machine has nothing to do. The fixed costs paid per `search` call are the same whether nq is 1 or 10,000: - **Host-to-device transfer** of the query matrix across PCIe. - **Kernel launch overhead** for each stage of the search. - **Device-to-host transfer** of the resulting distances and ids. - **Synchronisation** at the end of the call. At nq = 1 those costs are the entire runtime. At nq = 10,000 they are amortised over ten thousand searches and effectively vanish. This is why benchmark numbers for GPU FAISS are almost always reported as queries per second on a large batch — and why reproducing them with a one-query loop produces a confusing result. ## What this means for a real service Two workload shapes, two conclusions. **Batch or offline workloads** — deduplication, bulk semantic matching, scoring a corpus against a corpus, nightly recomputation — are the natural fit. You already have millions of queries, you can pass them in blocks of tens of thousands, and the GPU delivers its full advantage. **Online single-query serving** with a tight p99 is the awkward case. Each user request is one query, so the naive implementation hits exactly the overhead-dominated regime described above. Three ways out: 1. **Dynamic batching.** Put a small queue in front of the index: accumulate queries for a few milliseconds or until a size threshold, dispatch one batched `search`, scatter the results back. You spend a bounded amount of latency and gain a large multiple in throughput. This is the standard pattern and it is the answer interviewers are usually listening for. 2. **Accept CPU serving.** A well-tuned CPU IVF index answers a single query in a millisecond or two. If your QPS is modest and latency is sacred, the GPU may simply not be worth its cost and complexity. 3. **Split roles.** Use the GPU where it dominates — training the coarse quantizer, building the index, offline evaluation sweeps — and serve from CPU. This is a very common production shape and it sidesteps the batching problem entirely. ## Sizing the batch Batch size is not free: search allocates transient workspace proportional to the number of queries, the lists probed and k, so an enormous batch can exhaust device memory. The practical method is to sweep — measure queries per second at 1, 32, 256, 1024, 8192 — and you will typically see throughput climb steeply and then flatten. Pick a size just past the knee, verify the memory headroom under concurrency, and cap there. Going far beyond the knee buys nothing and risks an out-of-memory failure under load. ## Interaction with replication If you replicated the index across several GPUs, batching is not optional — it is the mechanism by which replication works. FAISS splits an incoming batch across the replicas, so a batch of one keeps a single device busy and leaves the rest idle. A four-GPU replicated deployment fed one query at a time delivers roughly the throughput of one GPU, at four times the hardware cost. ## How to answer this in an interview Start with the mechanism — fixed per-call cost versus parallel work available — rather than the slogan "GPUs like batches". Then convert it into a design: name dynamic batching as the serving pattern, state the latency-for-throughput trade explicitly, mention that batch size is bounded by device memory, and note the honest case where CPU serving is the better engineering answer. That progression from mechanism to tradeoff to decision is what the question is testing.
- How do you pick a batch size for FAISS GPU search?Sweep it. Measure queries per second at 1, 32, 256, 1024 and 8192 queries per call; throughput climbs steeply and then flattens. Choose a size just past the knee, then check device memory headroom at that size under production concurrency, because search workspace grows with batch size, probed lists and k. Going far beyond the knee adds memory risk for no throughput.
- Does batching help if the index is replicated across several GPUs?It is what makes replication work at all. FAISS splits an incoming batch across the replicas, so a batch of one occupies a single device and leaves the others idle. A four-GPU replicated deployment fed single queries delivers roughly one GPU's throughput at four times the cost — the batching layer is part of the design, not a later optimisation.
- When is serving from CPU the better engineering answer despite having GPUs available?When QPS is modest and the latency budget is tight. A tuned CPU inverted-file index answers a single query in a millisecond or two with no transfer or launch overhead and no batching queue to build and operate. The common compromise is to use the GPU for training and index building, where its advantage is largest, and serve from CPU.
saying these in an interview costs you the question
- Benchmarking GPU FAISS with a one-query-per-call loop
- Assuming a GPU lowers single-query latency versus a tuned CPU index
- Thinking bigger batches are always better regardless of memory
- Expecting replicated GPUs to help without a batching layer
- Treating throughput and latency as the same measurement