Search & Indexing

A batch rebuild periodically re-reads the whole catalog and builds a fresh index, so results lag by up to the rebuild interval. Near-real-time indexing applies each change as it commits, giving seconds of lag but needing an always-running pipeline.

A coordinator sends the query to every shard in parallel. Each shard scores its own documents and returns only its local top 10. The coordinator merges those lists and keeps the best 10 overall.

The URL frontier holds every discovered but unfetched URL and decides what to fetch next and when: it orders URLs by priority and spaces out requests to each host, so the crawler stays polite and spends its budget on valuable pages.

A top-k trie gives lookups proportional to prefix length but costs memory; an edge-ngram index reuses search infrastructure and ranks at query time, costing index size and latency; an FST is compact and fast but immutable, rebuilt offline.

The two writes share no transaction: a crash or timeout between them leaves one side stale, and concurrent writers can reach the index out of order. Deriving index updates from committed database changes, through an outbox or change capture, closes the gap.

Priority front queues feed back queues that each hold one host, pulled with a bias toward high priority. A heap keyed by each host's next allowed fetch time picks the host to serve, so no host is hit concurrently or too soon.

If each shard is slow 1% of the time, a query that waits on 100 shards is slow about 63% of the time. A hedged request sends a backup copy to another replica once the first is late and uses whichever reply arrives first.

Short prefixes are few, most typed queries pass through them, and their suggestion lists change slowly, so a small cache near users gets a high hit rate and removes both server work and much of the network round trip.

Give every event a monotonically increasing version from the source of truth, such as a row version or commit-log position, and write to the index only if that version is newer than the stored one. Partitioning events by product ID keeps each product's changes in order.

Any shard could supply results 9,991-10,000, so each shard must return its own top 10,000, and the coordinator merges 500,000 entries to keep 10. Cursor paging avoids this by resuming after the last hit's sort key.

Normalisation rewrites each URL into one canonical spelling (lowercase scheme and host, no default port or fragment, resolved dot segments, no session parameters); the crawler then checks a fingerprint of that form against a seen-URL set before admitting it to the frontier.

Keep the batch-built base and add a streaming path: count recent queries per short window, flag terms far above their own baseline, filter them for safety, and write them to a small TTL overlay merged at serving time.

Define lag as the time from a change committing in the database to it appearing in search results, and target a percentile, such as 99% of changes within 30 seconds. Measure it with commit timestamps carried through the pipeline plus synthetic canary products.

Record the change stream's position before the bulk load starts, load a snapshot into the shadow index, then replay every change from that position with version-guarded writes until it catches up. Only then switch the alias; keep the old index for rollback.

An exact hash such as SHA-256 changes completely when one byte changes, like a timestamp or an ad. SimHash or MinHash fingerprints stay close for similar pages, so near-duplicates show up as a small Hamming distance or a high estimated Jaccard similarity.

Assume no host deserves unlimited pages: cap each host's page budget and each URL's length and depth, flag repeating path segments and session-ID parameters, and watch for hosts whose many URLs share content fingerprints. Suspect hosts get lower priority or are cut off.

Hash-sharding a flattened prefix-to-list map spreads load evenly and routes each lookup to one shard but blocks traversal features like fuzzy matching; range-sharding a trie keeps traversal but needs load-based split points and special handling for short prefixes.

Return partial results only if the missing shards hold little of the likely answer and the consumer can tolerate gaps. Flag the response as partial, keep it out of the result cache, and fail instead where completeness is the requirement.

A document-partitioned index gives each shard a complete index over some of the documents, so every query hits every shard. A term-partitioned index gives each shard whole postings lists for some of the terms, so a query hits only its terms' shards.

Allow a small, length-dependent edit distance, walk the suggestion structure with an automaton that prunes branches exceeding it, rank exact matches above fuzzy ones, and cap the work, often using fuzzy matching only when exact results are too few.

Estimate each page's change rate from past fetches and weight it by importance, then spend the budget where a revisit most improves freshness. Revisiting strictly in proportion to change rate can backfire on pages that change faster than you can return.