How would you turn RAGFlow's retrieval-testing panel into a real evaluation loop?

## What the panel gives you, and what it does not The knowledge base's retrieval-testing panel runs a query against that knowledge base with an explicit similarity threshold, keyword weight and optional rerank model, and shows the chunks that came back with their scores and source documents. That is a genuinely good instrument: it isolates retrieval from generation and it exposes the numbers that drive the ranking. What it does not give you is memory. Each run is a spot check, so a team using it casually learns whether one query works today and nothing about whether last week's queries still work. The gap between the two is the entire evaluation story an interviewer is listening for. ## Build the question set from traffic, not imagination Harvest real questions: the ones users actually asked, especially the ones that failed. For each, record the ground truth as the document and passage that answers it. Include deliberate negatives — questions the corpus genuinely cannot answer — because a configuration that answers those is worse than one that returns nothing. Keep the set small enough to replay by hand at first; fifty well-chosen questions beats a thousand nobody runs. ## Replay on the changes that actually move retrieval The triggers are not only the obvious knobs. Threshold, keyword weight, candidate pool and rerank model change ranking directly. A chunk-template change or a document re-parse changes the chunks themselves, so previously good results can vanish while every setting looks untouched. An embedding-model change invalidates everything. Ingesting a large new batch shifts the score distribution and can push a previously safe chunk under the threshold. Every one of those deserves a replay. ## Sweep one variable at a time The discipline that separates evaluation from fiddling. Hold everything fixed, move the threshold across a few values, and record the outcome per question. Then reset and do the weight. Then decide on the reranker as a separate question, because it changes both the ranking and the score scale, so any threshold you had tuned must be re-tuned after it goes on. Record the whole configuration with each result set, or you will have numbers you cannot attribute. ## What to record per question Whether the known-good chunk was returned at all; at what rank; at what score; and how close that score sat to the threshold. That last number is the one teams skip and the one that predicts future breakage — a chunk clearing the floor by a hair will fall below it on the next re-parse. Reading the term-similarity and vector-similarity components separately also tells you which signal is carrying the query, which is what makes weight tuning evidence-based rather than superstitious. ## Keep retrieval and answer quality separate A good answer produced from a chunk that barely survived is a fragile pass, and a bad answer from a perfectly retrieved chunk is a prompt or model problem that no retrieval tuning will fix. Evaluating the two stages together hides both. Grade retrieval first on the panel's evidence, then grade answers only on questions whose retrieval already passed. ## Scaling past the panel At some point clicking stops being viable, and the loop moves to driving the same retrieval path programmatically with the same parameters, so the set can run in CI or on a schedule. The panel stays as the human debugging surface for the individual failures the automated run flags. The governance layer is the last piece: configuration changes reviewed rather than applied live, a replay required before any knowledge base is re-parsed in production, and an owner for the question set — otherwise it rots and everyone quietly stops trusting it.