How do you load millions of points into Qdrant without timeouts or duplicates?

## Why the naive version fails The obvious code — build a list of a million `PointStruct`s and pass it to one `upsert` — fails in three ways at once. It holds the entire dataset in client memory; it serializes one enormous request body; and it produces a single call that will hit a request timeout, after which you have no idea how much of the batch was applied. Bulk loading is therefore about streaming, acknowledgement semantics and idempotency. ## The batching helpers The Python client provides two upload calls that exist precisely for this. `client.upload_points(collection_name, points=iterable, batch_size=..., parallel=...)` accepts an **iterable or generator** of `PointStruct` objects and splits it into requests of `batch_size` points. Because it consumes lazily, you can stream straight from a file, a database cursor or an embedding pipeline without materializing everything. `client.upload_collection(collection_name, vectors=..., payload=..., ids=...)` takes parallel sequences — including a numpy array of vectors — which suits an offline job that already has embeddings in a matrix. Both take `parallel`, which spreads the work across worker processes. That helps when the bottleneck is client-side serialization rather than the server, but the data being sent must be picklable, and raising it beyond the server's ability to absorb writes just converts throughput into queueing. Start at 2 to 4 and measure. Batch size is a tradeoff between per-request overhead and blast radius: too small and you pay round-trip cost per point; too large and a single failure or timeout costs you a lot of redone work. Something in the low hundreds is a sane starting point for typical embedding sizes, tuned by watching request latency. ## wait: acknowledged versus applied Every write operation accepts a `wait` flag, and understanding it is the crux of this question. With `wait=False`, the server responds as soon as the operation has been **accepted** — written to the write-ahead log and queued for application. The call returns quickly and the write is durable, but the points are not guaranteed to be visible to a search issued immediately afterwards. With `wait=True`, the response comes only once the operation has been **applied** and the points are searchable. For a bulk load, `wait=False` is what you want: it lets the server pipeline application of writes while the client keeps streaming, and it stops slow segment work from stalling the loader. The Python client's `upsert` defaults to waiting, so this is an explicit choice you make, not a default you inherit. The rule of thumb is to load with `wait=False` and then, before any verification step or before flipping traffic over, do a waiting write or poll the collection until counts and status settle. Tests that write with `wait=False` and immediately assert on search results are a classic flaky-test source. ## Idempotency is the safety net Because `upsert` is keyed by point id and replaces rather than appends, retrying a batch that timed out is safe: the ids that already landed are simply overwritten with identical content, and the ones that did not are created. There is no duplicate-key error to handle and no compensating delete to write. This property is only real if your ids are **deterministic functions of the source data**. Derive them from the document's stable key — for instance a UUIDv5 over `"<doc_id>#<chunk_index>"` — and a resumed or re-run job converges on the same collection. Generate a random UUID per point instead, and every retry doubles part of your corpus with near-identical vectors that all match the same queries. This is the single most common way a bulk load quietly corrupts relevance. The same reasoning makes the job restartable: because re-writing a point is harmless, a crashed loader can restart from the last checkpointed offset — or even from the beginning — without a cleanup step. ## Transport and verification At this volume, transport matters. `QdrantClient(url=..., prefer_grpc=True)` uses the gRPC interface, which serializes large float arrays far more efficiently than JSON over REST and measurably raises sustained ingest rate. Also set a generous client `timeout` — the default is tuned for interactive queries, not for a batch that the server may take seconds to accept under load. When the load finishes, verify rather than assume. `client.count(collection_name, exact=True)` gives the exact point count to compare against your source. `client.get_collection(name)` returns the collection status along with `points_count` and `indexed_vectors_count`; a collection that is still working through the ingest reports a non-green status, and search served during that window may have lower recall than the finished state. Sampling a few known ids with `client.retrieve` confirms that payloads landed in the shape you expected — a check that catches serialization bugs no counter will. ## Failure modes to name in an interview Wrong-dimension vectors reject the entire request, so a single bad row can fail a whole batch — validate shapes before sending or narrow the batch to find the offender. Payloads that embed whole documents inflate every request and every later search response. And a loader that logs nothing per batch leaves you unable to answer "where did it stop", which is the question you will actually be asked at 3am.