How do you add an index to a huge MongoDB collection on a live replica set safely?

## Start with what a plain build already does The first thing to say is that the old foreground/background distinction is gone. From MongoDB 4.2 onwards, `createIndex` uses a single build process that holds an exclusive collection lock only **at the start and at the end** of the build. In between, it yields, so reads and writes against the collection continue; writes that arrive during the build are captured and applied to the new index before it commits. The `background: true` option that older tutorials tell you to pass is obsolete. The build is also a replica-set-wide operation. It starts on every data-bearing member at roughly the same time, and the primary does not commit the index until a **commit quorum** of members reports that their build finished. The default quorum is `votingMembers`, and you can override it per build: ```javascript db.orders.createIndex({ customerId: 1, createdAt: -1 }, { name: "cust_created" }, "votingMembers") ``` The practical benefit is that a failure — a duplicate key on a unique index, out of disk on one node — aborts the build everywhere rather than leaving members with different index sets. ## What it still costs No lock does not mean no impact. A build over a large collection reads the whole collection, sorts keys with a bounded amount of memory (governed by the `maxIndexBuildMemoryUsageMegabytes` server parameter, spilling to disk beyond it), and writes a new index structure. That is sustained CPU and I/O on every member at once, plus cache pressure that evicts hot documents. On a cluster already near its ceiling, that can be enough to push p99 latency past your SLO even though nothing is blocked. You can watch progress with `db.currentOp()` filtered to index builds, and abort a build with `killOp` if it is hurting. ## The rolling build When you cannot afford that load on the serving members, the rolling procedure keeps it off all but one node at a time. In outline, for each secondary: 1. Stop the member and restart it as a **standalone** — no replica set name, on a different port so application traffic and other members cannot reach it. 2. Build the index locally. Nothing is serving from this node, so the build can go as hard as the hardware allows. 3. Shut it down and restart it as a normal member of the set on its usual port. 4. Wait for it to catch up from the oplog and return to a healthy state before touching the next member. When every secondary carries the index, you `rs.stepDown()` the primary, wait for the new primary to be elected, and repeat the procedure on the demoted node. ## The risks you must name An interviewer is listening for whether you understand what the procedure gives up: - **Oplog window.** While a member is standalone, it is not replicating. If the build takes longer than the primary's oplog retains, the member cannot catch up when it rejoins and needs a full initial resync — which is far more expensive than the build was. Size the oplog against the expected build time before you start, and check the window between steps. - **Reduced redundancy.** With one member out of the set, you are one failure away from losing majority. On a three-member set that is a real exposure for the whole duration; do it during a low-traffic window and never take down two members. - **A step-down is a disruption.** The final step forces an election, so in-flight writes see errors and clients reconnect. Retryable writes absorb most of it, but it is not free. - **Sharded clusters need the balancer stopped.** A chunk migration in the middle of a rolling build moves documents between shards while some members have the index and others do not. Disable the balancer for the duration and re-enable it afterwards. - **Capacity while a member is out.** The remaining members carry the read and oplog load of the missing one. ## The cost that outlives the build The build is a one-time event; the index is forever. Every additional index is another structure written on each insert, and on each update that touches its key fields, plus index pages competing with documents for cache and a longer initial sync for any member you rebuild later. That is why the build conversation belongs with an index-review conversation: before you spend a rolling build, confirm from `explain` that the new index changes the plan you care about, and look for an existing index it makes redundant — the cheapest index to build is the one you do not need because a compound index already covers its prefix.