What does num_repetitions do in LangSmith's evaluate(), and what does it cost?

## The mechanic `num_repetitions` is an argument on `evaluate` and `aevaluate`. Set it to N and every example in `data` is executed N times, all inside **one** experiment. The runs are separate — each is traced individually with its own output, latency and scores — but they belong to the same experiment record, so the experiment's aggregate is computed across all of them. This is deliberately different from running `evaluate` N times, which would give you N experiments to eyeball side by side. One experiment with repetitions keeps the repeats together as one unit of measurement. ## Why anyone bothers A single execution per example tells you what happened once. LLM applications are stochastic: sampling temperature above zero obviously, but also load-dependent routing, provider-side model updates, and any nondeterminism in retrieval. At `num_repetitions=1`, an example that passes 60% of the time and one that passes 100% of the time look identical in a green run — and the flaky one is the one that will page you. Repetitions convert that invisible property into a visible one. Three executions of the same input let you see agreement: three passes, or two passes and a fail. Aggregated over the dataset, this is the difference between "the suite scored 0.82" and "the suite scored 0.82 and here are the eleven examples that were not stable across repeats". The unstable rows are usually the interesting engineering work — an under-specified prompt, an ambiguous reference, a retrieval step whose ranking flips. ## What it costs Linear, in every dimension, and the multiplication catches people out because it applies to the evaluators too: - target calls: `examples × N` - LLM evaluator calls: `examples × evaluators × N` - wall clock: roughly `× N` at fixed concurrency - storage and trace volume: `× N` A 500-example dataset with two judge metrics is 1,500 model calls at N=1 and 6,000 at N=4. If the judge is an expensive model, the judge line dominates the bill, which is an argument for a cheaper judge before it is an argument against repetitions. ## Where to spend it The usual policy is: repetitions off for the frequent, fast loop, and on for a small deliberate subset. Concretely, a few dozen examples at N=5 costs less than a few hundred at N=1 and tells you something the big cheap run cannot. When you are specifically investigating stability — a new sampling setting, a model swap, a prompt that reviewers say "sometimes" misbehaves — that is the run to make. It is also a diagnostic tool for the eval suite itself. If a suite goes red intermittently and you cannot tell whether the application or the judge is unstable, repetitions separate them: an example whose target output is identical across repeats but whose score moves has an unstable *judge*, not an unstable application. That is a genuinely useful thing to be able to demonstrate. ## What it does not give you Repetitions measure spread. They do not, by themselves, tell you whether the gap between two experiments is real — that is a statistical question about sample size and effect size, and it lives with evaluation methodology rather than with the tool. What the tool gives you is the raw material: repeated observations per example, each individually traceable. They also do not fix a fundamentally unstable system. If half your examples flap, the answer is to make the application more deterministic — constrain the output format, pin the model version, tighten the prompt — not to average over more samples until the number looks calm. Averaging a flapping system produces a stable number describing an unstable product. ## Interview shape A good answer names the mechanic (N executions per example, one experiment), names the cost multiplier including evaluators, and says where you would actually spend it: small subset, high N, when the question is stability. Mentioning the judge-versus-application diagnostic is the detail that reads as having actually used it.