What does Qdrant's optimizer indexing_threshold do, and why lower it during bulk load?

## The segment model A Qdrant collection is physically a set of segments. New points land in a fresh segment; a background optimizer merges small segments into larger ones, vacuums deleted points, and builds vector indexes. `models.OptimizersConfigDiff` is how you influence that machinery, on `create_collection` or `update_collection`. ## What indexing_threshold means `indexing_threshold` is a size in kilobytes of vector data. A segment below it is left unindexed and served by exhaustive scan; above it, the optimizer builds an HNSW graph for that segment. This is not a limitation — it is the correct trade. Brute-force scanning a few thousand vectors is fast, exact, and free of build cost; an HNSW graph over the same data costs CPU to build, adds memory, and returns approximate results. The threshold is simply where the crossover sits. The default is in the tens of megabytes of vector data, which for typical embedding sizes corresponds to tens of thousands of points. ## Why bulk loading fights it During a large ingest, segments are constantly being created, crossing the threshold, being indexed, then being merged into bigger segments — which invalidates the graph just built and triggers another build. A meaningful fraction of ingest-time CPU is spent producing indexes that are discarded minutes later, and that CPU is competing with the write path. The remedy is to tell the optimizer not to bother until the data has settled: 1. `client.update_collection(collection_name="docs", optimizers_config=models.OptimizersConfigDiff(indexing_threshold=0))` — zero disables index building. 2. Upload the data. 3. `client.update_collection(collection_name="docs", optimizers_config=models.OptimizersConfigDiff(indexing_threshold=20000))` — restore, and the optimizer now builds graphs once, over merged segments. During step 2 the collection remains queryable, but every search is a full scan, so latency will be poor and will degrade as data grows. That is the trade you are explicitly accepting. ## Knowing when it is done After restoring the threshold, indexing runs in the background. `client.get_collection(collection_name="docs")` reports the collection's status and counts — including how many points are actually indexed versus merely stored. Benchmarking before that settles produces numbers that describe a half-built index and are worthless. The single most common false alarm in Qdrant operations is "queries are slow after our import" measured while the optimizer is still working. ## The neighbouring knobs `default_segment_number` controls how many segments the optimizer aims for; 0 lets Qdrant choose based on CPU count. Fewer, larger segments give better search performance because a query fans out across segments and merges results — but they are more expensive to optimize and rebuild. More, smaller segments favour write throughput and parallel optimization. `memmap_threshold` is a size in kilobytes above which segment data is memory-mapped rather than held in RAM. It is a memory-pressure control that overlaps with the `on_disk` flags on vectors and on the HNSW graph; on modern versions setting storage explicitly with those flags is the clearer way to express intent. `max_segment_size` caps how large a segment may grow, `flush_interval_sec` governs how often data is flushed, and `deleted_threshold` / `vacuum_min_vector_number` decide when a segment carrying many deletions is rebuilt to reclaim space. That last pair matters for workloads that churn: heavy deletion without vacuuming leaves tombstoned points occupying memory and being traversed. ## Operational guidance Treat these as deployment configuration, not tuning fiddles. Set them deliberately for the collection's workload shape: a write-heavy pipeline wants more segments and a suppressed indexing threshold during loads; a read-heavy serving collection wants fewer, larger, fully indexed segments. Change them through `update_collection` during quiet periods, because every change hands the optimizer more background work, and always confirm the collection has returned to a settled state before drawing conclusions from latency measurements.