How do you move a FAISS index onto a GPU, and what does StandardGpuResources do?

## What the GPU side of FAISS actually is FAISS is a library, not a server, and its GPU support is a parallel set of index implementations that live in device (GPU) memory. You do not point a config file at a GPU; you either build a GPU index class directly or you clone a CPU index onto a device. Nothing about the search API changes — the same `train`, `add`, `search` calls apply — but the data now lives in GPU RAM and the distance computations run as CUDA kernels. ## StandardGpuResources `faiss.StandardGpuResources()` is the handle for one GPU's runtime state. It holds: - a **temporary scratch pool** pre-allocated on the device, reused by every search so kernels do not hit `cudaMalloc` on the hot path (device allocation is slow and synchronising); - the CUDA **stream** the index's kernels are enqueued on; - the **cuBLAS handle** used for the matrix multiplications behind distance computation. Because the pool is reserved up front, GPU memory usage jumps the moment you create the resources object, before you have added a single vector. You can shrink it with `res.setTempMemory(n_bytes)` or disable it with `res.noTempMemory()` (which trades memory for slower, allocation-heavy searches). One resources object per GPU is the normal pattern; sharing one across many indexes on the same device is fine and preferred, since they then share the same pool. ## Cloning an index onto a device ```python res = faiss.StandardGpuResources() gpu_index = faiss.index_cpu_to_gpu(res, 0, cpu_index) ``` The third argument is the CPU index; the second is the device ordinal (`faiss.get_num_gpus()` tells you how many are visible). The clone copies the index's contents into device memory — so the CPU copy still exists, and you now hold two copies of the data until you drop one. For a multi-GPU spread there is `faiss.index_cpu_to_all_gpus(cpu_index, co=..., ngpu=...)`. You can also skip the CPU step and construct a GPU index directly, e.g. `faiss.GpuIndexIVFFlat(res, d, nlist, faiss.METRIC_L2, config)`, which avoids ever materialising a CPU copy. That matters when the dataset only just fits: cloning needs both copies resident at the moment of the copy. ## The reference-lifetime trap The SWIG-generated Python wrapper does **not** take ownership of the resources object when you pass it in. If `res` is a local variable in a function that returns the index, Python garbage-collects it, the C++ index keeps a dangling pointer, and the next search crashes the interpreter — usually with no useful traceback, which is why this bug eats hours. Store `res` alongside the index (an attribute on the object that owns the index is the simplest fix). The same rule applies to the list of resources you build for a multi-GPU setup. ## Getting data back off the GPU `faiss.index_gpu_to_cpu(gpu_index)` produces an ordinary CPU index with the same contents. This is required before `faiss.write_index(...)`, because serialisation is defined for CPU index types only. The practical deployment pattern that follows: build or train on GPU (which is where the speedup is largest), convert down, write the file, and then let each serving replica decide whether to load it on CPU or clone it back onto its own device. It also gives you a fallback — the same artifact serves on a CPU-only host if a GPU box is unavailable. ## What to say in an interview Name the three moving parts — resources object, `index_cpu_to_gpu`, `index_gpu_to_cpu` — and then show operational awareness: the scratch pool is pre-reserved, the resources object must outlive the index, and persistence goes through the CPU form. Candidates who have only read the README describe the clone call and stop there.