How would you prove a chunking change actually improved a LlamaIndex RAG pipeline?

## Why "it feels better" is not an answer Chunk size, overlap, splitter choice, top-k and reranking all interact, and every one of them changes the text that reaches the model. Without a fixed measurement, a team cycles through configurations forever, each change validated by whoever tried three questions they remembered. The interview question behind this is whether you can design an experiment, not whether you can name a metric. ## Step one: build the eval set, and treat it as an asset You need two kinds of labels, and they are not the same. For retrieval you need question → correct-node pairs. `generate_question_context_pairs(nodes, llm=..., num_questions_per_chunk=2)` synthesises these by asking an LLM to write questions answerable from each node, producing an `EmbeddingQAFinetuneDataset` that maps query ids to the node ids that should be retrieved. Synthetic generation gets you to a usable set in an hour; a human pass to delete degenerate questions ("what does this section say?") is what makes it trustworthy. The known weakness is that synthetic questions inherit the chunking used to generate them, which biases comparisons between chunkings — mitigate by generating from the raw documents or by keeping node-level labels aligned to source spans rather than to one splitter's output. For generation you need question → reference answer pairs, which is what `CorrectnessEvaluator` consumes. These are more expensive to produce and more valuable; they are also the set real stakeholders will read when they challenge a result. Version both sets in the repo. An eval set that quietly changes between runs destroys every comparison you have made. ## Step two: measure retrieval independently `RetrieverEvaluator.from_metric_names(["hit_rate", "mrr"], retriever=retriever)` builds an evaluator over a retriever; `aevaluate_dataset(qa_dataset)` runs it across the labelled set. Hit rate answers "was the right node retrieved at all", MRR answers "how high did it rank". This half is cheap — embeddings and vector search, no judge LLM — so run it first and run it often. The reason to isolate this half is causal clarity. Chunking changes what can possibly be retrieved. If hit rate is flat between two chunk sizes, then the retrieved evidence is effectively unchanged, and any wobble in downstream answer scores is judge variance rather than an improvement. If hit rate rose but answer quality did not, the bottleneck is synthesis, and the next experiment belongs there. ## Step three: measure generation Run `BatchEvalRunner` with the faithfulness, relevancy and correctness evaluators over the same questions, end-to-end through the query engine. Keep the per-row results, not just the aggregate: the interesting output is the set of questions whose verdict flipped between configurations, and the judge's `feedback` on them. Ten flipped rows read carefully teach more than a two-decimal aggregate. ## Step four: control the experiment One variable at a time. Same eval set, same judge model, judge temperature 0, same generator and same top-k when chunking is what you are testing. Record the full configuration with the result — chunk size, overlap, splitter, embedding model, top-k, reranker, generator, judge. A result without its configuration is not reproducible, and six weeks later nobody remembers which run had reranking on. Judge drift deserves specific attention: when the judge model version changes, historical numbers are no longer comparable to new ones. Either pin the judge or re-baseline the whole history when you move it. ## Step five: report cost and latency next to quality Smaller chunks usually improve hit rate, because a precise chunk embeds more cleanly — but they also push you toward a higher top-k to retain context, which inflates prompt tokens linearly. Token counting gives you that number. A configuration that gains three points of faithfulness while doubling prompt cost and adding 400ms to p95 is a tradeoff for a product owner to make, and presenting the quality number alone is quietly dishonest. ## Step six: respect statistical power Small eval sets plus a nondeterministic judge means small deltas are noise. Before believing a result, ask what effect size the set can detect: on fifty questions, one flipped answer is two points. The practical rules are to grow the set for decisions that matter, repeat borderline runs to see the spread, and prefer paired comparison — the same questions under both configurations, looking at which rows flipped — over comparing two independent aggregates. ## Making it durable Once this exists, wire it in: the cheap retrieval half on every change to indexing code, the expensive generation half on release candidates and nightly. Publish the numbers where the team sees them. The value is less any single measurement than the fact that the next chunking argument gets settled by a run rather than by seniority.