Your 30M-chunk vector index no longer fits RAM — what do you change?

## Do the arithmetic before choosing anything The memory bill is not mysterious. Vectors cost `chunks * dimensions * bytes_per_component`. Thirty million chunks of 1024-dimensional float32 is roughly 120 GB. A graph index adds roughly `8 * m` bytes per vector for its links — about 8 GB at m=32 — plus payloads and identifiers. Any answer that starts with an index parameter rather than this calculation is answering the wrong question, because no HNSW tuning changes the leading term. The second number to establish is the requirement: what recall does this workload actually need, at what p95 latency, at what query rate, and how often does the corpus change? A legal e-discovery matter under a production deadline has a defensibility obligation that makes recall close to non-negotiable and tolerates a slower query; a consumer type-ahead is the reverse. These two numbers — the bill and the requirement — frame every lever below. ## Lever 1: fewer dimensions The cheapest reduction is to store shorter vectors. Several current embedding models are trained so that a prefix of the vector is itself a usable embedding, which means you can truncate — say 1024 down to 512 or 256 — and lose less quality than the size reduction would suggest. Halving dimensions halves the leading term and speeds every distance computation. It requires re-embedding or re-slicing and a recall re-measurement, so validate on a labelled set before committing. ## Lever 2: quantization plus rescoring Quantization stores each component in fewer bits. Scalar quantization to int8 is about 4x smaller; binary quantization, one bit per dimension, is about 32x smaller and turns distance computation into extremely fast bitwise operations. Used naively, aggressive quantization loses meaningful recall. Used properly, it is paired with **rescoring**: search the quantized index for an oversampled candidate set — several times k — then recompute exact distances for those candidates against full-precision vectors held on disk or in a cheaper tier, and take the true top-k. The compressed index does the navigation; the full vectors do the final ordering. This pattern is the workhorse of large deployments because it keeps the hot structure small while preserving most of the accuracy, at the cost of one extra read-and-rescore step per query. ## Lever 3: disk-resident index designs A graph index whose working set exceeds RAM does not degrade gracefully — every hop becomes a random page fault. Indexes designed for disk residency, in the DiskANN family and its Postgres implementations such as pgvectorscale, invert the layout deliberately: compressed vectors stay in memory to guide the search, while the graph and full-precision vectors live on SSD, and the graph is built so that a query needs only a small number of reads. The result is a dramatically smaller memory footprint at a latency cost measured in milliseconds rather than the catastrophic cliff of swapping. This is the standard answer when the corpus genuinely exceeds affordable RAM and recall must stay high. ## Lever 4: shard along a boundary queries never cross In many domains there is a partition key that no legitimate query spans — one legal matter, one tenant, one fiscal year. Sharding on it means no single index must be whole: hot shards stay resident, cold shards live in object storage and are loaded on demand. Object-storage-native and multi-tenant-first designs make this the primary architecture rather than an afterthought, trading a cold-start penalty on a first query for very cheap storage of a long tail. It also simplifies filtering, since the partition key stops being a predicate the index must handle, and it makes deletion of an entire tenant or matter a file operation. The cost is many indexes to operate and a routing layer that must be correct. ## Lever 5: question whether you need vectors at all As of mid-2026 this is a live option rather than a provocation. When an agent can issue iterative keyword searches against the source — grepping a repository, querying an existing full-text index — it can refine its own queries across turns in a way a single embedding lookup cannot. Reported results have put agentic keyword search close to vector retrieval on answer faithfulness for some corpora, and at least one prominent coding tool dropped its local vector index in favour of it. The trade is more model calls and higher per-query latency against no embedding pipeline, no index to host, no reindexing on every document change, and no stale-embedding problem. For corpora that change constantly, or where the queries are naturally lexical, that trade can be strongly favourable. For large heterogeneous prose corpora where paraphrase matching matters, it is usually not. ## Deciding Order the levers by how much they cost you to reverse. Truncation and quantization are configuration and a re-index. A disk-resident index is a deployment change. Sharding is an architectural commitment that reaches into routing and lifecycle. Dropping vector search entirely is a product decision. Take them in that order, and after each one re-measure recall against exact search on the same labelled query set, because that is the quantity you are spending. The supporting choice — whether this lives in the Postgres you already operate or in a dedicated vector engine — should follow the numbers, not enthusiasm. Keeping vectors alongside the relational data you already back up, secure and monitor is worth a great deal, and Postgres extensions now cover both graph and disk-resident designs. A dedicated engine earns its operational cost when query volume, filtering sophistication or multi-tenant scale exceed what the general-purpose database is happy doing. Agent-driven workloads push in that direction, since agents issue far more queries per task than a human does. ## Where teams go wrong Buying more RAM as a reflex, which postpones the decision at increasing cost; quantizing without a rescoring pass and discovering the recall loss in production; sharding on a key that queries do turn out to cross; and — most common — never establishing the recall requirement, so no one can say whether any of these trades was acceptable.