An encoder-tuned passage ranks in your offline harness but never retrieves live — what explains the gap?
answer
- the harness proved something about your function
- chunks, not the document you scored
- extraction changed the bytes first
- presence before rank
- a bigger corpus is harder competition
basics
~20 sAn offline harness proves a property of a function you hold, not of a deployment. The live path differs in the unit indexed, the text extracted, the encoder pin, the scoring mix and the competition — any one ends the fit.
solid answer
~50 sStart by separating the two claims. Offline you showed that a string you wrote is near a query vector under an encoder you ran. Live retrieval requires much more of the pipeline to agree. The common divergences: the index holds chunks, so the vector that exists is of a fragment rather than the whole document you scored; ingestion extracted and normalised the text before embedding, so the bytes encoded were not your bytes; the deployed pin may not be the version you ran; first-stage scores may be fused with a lexical signal or narrowed by a predicate; and the real corpus dwarfs your local shard, so a comfortable rank now competes against work you never scored under a fixed budget. Before concluding it failed, establish whether it was indexed at all — that is a different bug from being outranked.
go deeper
Know that a retrieval index stores vectors of chunks, not of whole documents, and that ingestion transforms text before it is ever embedded.
Be able to list the stages between a submission and a candidate list — extraction, normalisation, chunking, embedding under a pinned model, scoring, budget — and name what each one can change.
Demonstrate the triage order out loud: establish presence before arguing about rank, then compare the unit and the text you scored against what the pipeline would have encoded. Say what your harness result actually claims.
Own the standard for what gets filed: a result with no recorded conditions and no rate is unfalsifiable later, and that costs a programme more than an unfiled finding does.
### Two different claims An offline scoring harness establishes one thing: under an encoder you ran, on text you controlled, a candidate string sat close to some query phrasings. A live retrieval is the conjunction of every stage between a submission and a candidate list. The gap between the two is where most of this family's findings die, and the triage question is not why it failed but which of the two claims you actually have. ### Where the live path diverges **The indexed unit is not your artefact.** You embedded a document; the index holds chunks. If ingestion split the submission at boundaries you did not model, the vector that exists is of a fragment, and the fitted whole never became a point in the space. **The bytes changed before encoding.** Ingestion parses, extracts and normalises. Text taken from an uploaded document is produced by an extractor, and whitespace collapsing, case folding, unusual-codepoint handling or truncation to a token budget can all alter exactly the material a fit depends on. **The pin is not the one you ran.** A published version is only what the project documents; a deployment may run a different one, or serve a differently configured pooling or normalisation. **The first stage is not the whole ranking.** Many deployments fuse dense similarity with a lexical score, or apply a metadata predicate that narrows the pool before anything is scored. A construction that wins on distance alone can be outside the pool entirely. **The competition is bigger.** A local shard flatters everything. In the real corpus the region around a common question is crowded with genuine work, and a fixed candidate budget means being close is not enough. **Timing.** Ingestion and index rebuilds are batched. A submission may simply not be embedded yet at the moment you tested. ### Ordering the checks The first split is presence versus rank, because the two have nothing in common as bugs. If any path returns the submission at all — a lexical query on a distinctive phrase, a metadata lookup, an id fetch where the interface allows it — then it is indexed and losing on score, and the fit is the suspect. If nothing returns it under any phrasing, suspect ingestion, extraction or timing, and stop tuning vectors. Only then is it worth comparing what you encoded against what the pipeline would have encoded: same unit, same text, same pin. ### Is the gap itself the finding Sometimes. If the honest conclusion is that a construction of this family reliably fails against a deployment whose chunking and scoring mix you cannot observe, that is a real result about the method's reach, and worth writing down as such. What is not a result is a single live hit obtained after enough variation: a probabilistic pipeline over a moving corpus will produce occasional hits, and one success against one deployment on one day is not a reproducible finding. If you file it, file the rate across trials and the conditions — the pin, the query set, the corpus state at the time — or the next person will read a later non-reproduction as a fix rather than as drift.
- How would you tell a not-indexed-yet failure from an indexed-but-outranked one?Ask for presence, not rank. Query on a distinctive literal phrase from the submission, or use any lexical or metadata path the interface exposes; if it comes back at all, it is indexed and losing on score. If nothing returns it under any phrasing, suspect ingestion, extraction or a batch window and stop adjusting the text. Where ingestion timestamps are visible in results, they date the index directly.
- One live retrieval in twenty attempts — do you file that as a finding?Not as it stands. It shows the construction worked once against one deployment, which a probabilistic pipeline over a changing corpus can produce by drift alone. File it with the rate across trials, the encoder pin, the query set and the corpus state, or state plainly that it is unreproduced. A finding whose conditions are not recorded cannot be distinguished later from noise, and the person triaging it will spend your credibility discovering that.
- Why does a local shard of the corpus flatter your result so badly?Because ranking is competitive and a candidate budget is fixed. Distance to the query is unchanged by corpus size, but the number of passages closer than yours is not. Genuine work on the same subject clusters in the same region, so a rank that looked comfortable against a few thousand chunks can fall far outside a top-k over millions.
saying these in an interview costs you the question
- Treats the document as the indexed unit when the index holds chunks
- Assumes the deployed encoder pin matches the published one
- Concludes the plant failed without checking whether it was ingested
- Reports an offline harness score as a live result
- Calls a single lucky retrieval a reproducible finding