How would you shard an approximate nearest-neighbour index of job postings across locales and private employer boards?

## Two partition keys, two different reasons Sharding questions go wrong when one key is asked to serve two purposes. State the driver first: | | locale partition | tenant partition | |---|---|---| | driver | cost and latency: smaller structures, fewer touched per request | correctness: a private board must not be reachable from outside | | chosen | by the seeker's commute area, at query time | by the request's tenant context, before any search runs | | cost of getting it wrong | lost recall for that request | disclosure of postings that should not have been visible | | sizing | balanced by posting volume, hot areas replicated | dictated by tenancy, so wildly uneven by nature | | may be relaxed | yes — fan out to more shards when the seeker is mobile | no | ## Locale as a performance partition Four million postings at 256 dimensions in single-precision floats is roughly four gigabytes of raw vectors — 256 times 4 bytes is about a kilobyte per posting. Split forty ways that is around a hundred megabytes per shard, small enough to sit in memory with the structure on top and cheap enough to rebuild independently. The gains are concrete: a request touches the one or two partitions its seeker cares about, rebuilds are per-partition rather than global, and a bad build blast-radiuses to one area. The complications are equally concrete: - **Mobile seekers.** Someone relocating legitimately wants two or three areas, so the router must be able to widen the fan-out, and depth then has to be divided across the partitions it touches. - **Remote-eligible postings** belong to every area and none. Either duplicate them into each partition, which multiplies their storage and their update cost, or keep a separate remote partition that is always included in the fan-out. The second is usually cleaner because the duplicate update path is where drift creeps in. - **Hot areas.** One metro can hold a fifth of the postings and a third of the traffic. That is a **replication** problem, not a re-partitioning problem: copy the hot partition, do not invent a finer key that makes routing harder for everyone. Because every partition holds vectors from the same encoder under the same metric, per-partition results are comparable and the tier merges them into one candidate set before filtering. The mechanics of fanning out and merging are ordinary distributed-query work; the retrieval-specific part is that depth is a budget being split, so a request touching three partitions must ask each for enough that the merged set still fills the shortlist. ## Tenancy as a correctness boundary Private employer boards — an internal mobility board, a staffing partner's exclusive inventory — carry a hard requirement: those postings are visible to a defined audience and to nobody else. Three tempting designs weaken it: 1. **A shared index with a tenant filter on the results.** Isolation now depends on every retrieval path applying the filter, including the new source somebody adds next quarter. It also spends the depth budget on candidates that will be discarded. 2. **Folding the tenant identifier into the vector** so similarity separates tenants. This makes a hard requirement statistical: nothing rules out a cross-tenant neighbour, it is merely unlikely. 3. **Leaving it to the scoring or list-editing stage.** That puts a disclosure boundary behind a ranking stage whose job is relevance, and every stage in between handles data it should never have held. The design that holds is routing on tenant **before** the search: the request's tenant context selects which index is queried at all, so a leak requires a routing bug rather than a missing predicate, and the isolation is auditable in one place. ## The long tail of tiny boards Per-tenant indexes create thousands of very small structures — a private board with two hundred postings is common. Two adjustments follow: - **Below a few thousand vectors, do not build an approximate structure at all.** Scanning two hundred vectors exactly is trivially cheap and gives perfect index recall; the approximate structure only adds build cost, memory overhead and a cold first query. - **Amortise the overhead**: small tenants can share a process and be loaded on demand, as long as the *search scope* stays per tenant. Sharing a host is fine; sharing an index is the thing that is not. ## Composing the two The order is fixed by the reasons: **tenant first, locale second.** A request resolves its tenant context, which selects either the public index — where locale routing then applies — or exactly one private board's index, where locale partitioning is usually pointless because the board is small. Writing it in the other order produces the shared-index-plus-filter design nobody wanted.