When a product-search index is fully rebuilt into a shadow index, how do you keep the writes that arrive during the rebuild?

## Why a full reindex happens Most index changes are incremental, but some force every document to be rewritten: - a field changes type, for example a text field that must become a numeric filter; - text analysis changes, so existing documents were prepared the old way; - the index is split into a different number of shards; - a new field must be populated for every product; - the index drifted or was damaged and must be rebuilt from the source of truth. For an e-commerce catalog the rebuild may take hours, and shoppers must keep searching throughout. ## The shadow index and the alias A **shadow index** is a new, empty index built with the new schema next to the **live** one. Readers never query an index by its concrete name; they query an **alias**, a stable name that points at exactly one index. Switching the alias from the live index to the shadow is a single metadata change, so queries move over at once and can be moved back just as quickly. The hard part is not the switch, but ensuring the shadow is complete and current at the moment of switching. ## The catch-up sequence 1. **Record the change stream's current position** (the offset in the log that feeds the indexer). 2. **Create the shadow index** with the new schema and write-friendly settings. 3. **Bulk-load a snapshot** of all products read from the database after step 1. 4. **Start a second consumer** that replays changes from the recorded position into the shadow, while the existing consumer keeps updating the live index. 5. **Wait for catch-up**: the shadow consumer's lag falls within the normal SLO. 6. **Verify**: compare document counts, sample queries against both indexes and compare results, and check error logs for rejected documents. 7. **Switch the alias** to the shadow, and make the shadow's consumer the live one. 8. **Keep the old index** for a rollback window, then delete it. ```pseudocode start_pos = change_log.current_position() shadow = create_index(new_schema) for batch in read_snapshot(source_db): // snapshot read begins after start_pos shadow.bulk_write_if_newer(batch) shadow_consumer = consume(change_log, from = start_pos, apply = shadow.write_if_newer) wait_until(shadow_consumer.lag() < lag_slo) if not verify(shadow, live): abort() // alias untouched, readers unaffected alias.point_to(shadow) ``` ## Why the order of steps 1 and 3 matters If the snapshot started **before** the position was recorded, changes committed between the two would be in neither the snapshot nor the replay, and the shadow would silently miss them. Recording the position first guarantees that every change is in the replay, the snapshot, or both. "Both" is the normal case, which is why every write must be **version-guarded**. A change can be in the snapshot and replayed later; the replayed copy carries a version equal to the stored one and is skipped. A replayed change can also be older than the snapshot row, and the guard rejects it. Deletes need versioned tombstones for the same reason. ## Two ways to handle in-flight writes | Approach | How it works | Trade-off | |---|---|---| | **Replay from a recorded position** | A second consumer reads the change log from the recorded position into the shadow | Live path untouched; needs log retention longer than the rebuild | | **Dual-feed during the rebuild** | The indexer writes every new change to both live and shadow while the bulk load runs | No long retention needed; the snapshot can overwrite newer dual-fed data unless writes are version-guarded, and the live indexer's code changes | Both approaches rely on the same versioning; replay is usually simpler to reason about because the live path does not change. ## Risks to plan for - **Log retention**: if the rebuild takes 10 hours and the change log keeps 7 days, there is room; if retention is shorter than the rebuild, replay starts from data that is already gone. - **Load**: the snapshot read and the bulk load compete with production traffic, so throttle both. - **Capacity**: two full indexes exist side by side, so plan roughly double the storage for the duration. - **Rollback**: keep the old index and the old consumer's position until the new one has served real traffic without problems.