How do you load ten million vectors into a Pinecone index efficiently?

## Where the time actually goes At ten million vectors, three costs compete and only one of them is Pinecone: 1. **Embedding** — usually the real bottleneck. Ten million texts through an embedding model is typically far more time and money than the writes. Any credible answer mentions that the ingest pipeline is embedding-bound, and that you checkpoint embeddings so a failed load never re-embeds. 2. **Network round trips** — one request per record is ten million sequential HTTPS calls. At even 30 ms each that is 83 hours. This is the number worth quoting because it makes the case for batching and concurrency instantly. 3. **Write throughput at the service** — bounded, and it pushes back with rate limiting when you exceed it. ## Lever one: batch Group records into `upsert` calls of about 100. The constraint is the request size ceiling of roughly 2 MB — a 1536-dimension float vector serialises to several kilobytes, so a couple of hundred records with modest metadata is near the limit and fat metadata gets there sooner. Batching turns ten million round trips into a hundred thousand. Size the batch by measurement, not folklore: serialise one representative record, divide the budget, take a safety margin. If you see request-too-large errors, halve the batch. ## Lever two: concurrency Batching alone still leaves you serial. The Python client supports in-flight parallelism: ``` index = pc.Index("docs", pool_threads=30) futures = [index.upsert(vectors=chunk, async_req=True) for chunk in chunks] results = [f.get() for f in futures] ``` `pool_threads` sizes the connection pool; `async_req=True` makes each `upsert` return a future instead of blocking. The critical discipline is **calling `.get()` on every future**: without it, failed batches vanish silently and you discover the hole weeks later as missing search results. Bound how many futures are outstanding at once (a sliding window rather than materialising ten million calls) so memory stays flat. If you drive ingestion from multiple worker processes, partition by id so two workers never write the same record, and remember there is no cross-record transaction — partial visibility during the load is normal. ## Lever three: backpressure Serverless indexes rate-limit writes. Getting 429s is not a bug, it is the system telling you the ceiling. The correct response is exponential backoff with jitter and a reduction in concurrency; the incorrect one is an immediate tight retry, which converts a slowdown into a thundering herd. A good loader adapts: raise concurrency while responses are clean, drop it on rejection. Also make failures loud — count them per batch, log the ids, and let the job exit non-zero rather than reporting success over a partial load. ## Idempotency and restartability A multi-hour load *will* be interrupted. Two properties make that survivable: - **Deterministic ids** (derived from the source, not generated fresh per run), so re-sending a batch overwrites rather than duplicates. - **A checkpoint** of which input shards have been acknowledged, so a restart resumes rather than re-runs. Combined, resuming is safe even if the checkpoint is slightly stale — the overlap is harmless overwrites. ## Verifying the load `describe_index_stats()` reports vector counts per namespace, but those counts lag while writes are in flight, so it is a **reconciliation** tool, not a progress bar. Track progress from your own acknowledged-batch counter. When writing has stopped and settled, compare your count against the index stats; a persistent shortfall means real dropped batches and warrants investigation, whereas a gap mid-load means nothing. Also remember: freshly written vectors are not instantly searchable. Do not kick off recall evaluation the moment the loader exits — let the index settle first. ## Alternatives worth naming - **Import from object storage.** For very large initial loads Pinecone provides a bulk import path that reads prepared files from object storage rather than having your client push every record. Where it fits, it removes client throughput from the equation entirely and is usually cheaper than a client-side load. It is an initial-load tool, not a substitute for streaming upserts of ongoing changes. - **Load into a fresh namespace and cut over.** Building a large corpus into a new namespace, then switching reads and dropping the old one, gives a clean rollback and avoids serving a half-built corpus. It also means a failed load costs you nothing but the namespace. - **Keep metadata small.** Every unnecessary metadata byte is multiplied by ten million on the way in and by `top_k` on every query afterwards. ## The judgment being tested An interviewer wants to hear that you sized batches from the request limit rather than guessing, that you parallelised with backpressure rather than open-loop, that failures are surfaced and the job is restartable, and that you know the write path is probably not the slowest part of the pipeline.