How should you report deep RL returns across seeds instead of showing the best run's curve?

## What is being asked A deep RL result is a *distribution* over runs, not a number. The reporting question is how to compress that distribution honestly, and how many samples of it you need before compressing at all. ## How many seeds The common practice of three seeds comes from compute cost, not statistics. With outcomes that can range from 0 to 3,100 for one unchanged configuration, three samples give a confidence interval wide enough to contain almost any claim, and the probability that all three land in the same mode is high enough to produce a confidently wrong conclusion in either direction. Ten is the usual working minimum for a claim of the form 'A beats B'; more if the two methods are close or if the outcome is bimodal. Fix the number in advance. Choosing when to stop after seeing the numbers is a way of selecting on noise. ## Which aggregate - **Mean.** Sensitive to both modes. With four failures and six successes it reports a value no run achieved, and one extra failure moves it a lot. - **Median.** Robust, but throws away most of the sample and is itself noisy at small seed counts. - **Interquartile mean (IQM).** Discards the top and bottom quarters of runs and averages the rest. It keeps half the data, so it is less noisy than the median, while remaining insensitive to a couple of collapsed or lucky seeds. This is the aggregate recommended by the recent literature on statistical practice in deep RL, together with performance profiles that show the whole distribution. - **Maximum.** Not an aggregate. See below. ## Which uncertainty band A mean plus or minus one standard deviation encodes an assumption of a symmetric, roughly normal spread. Deep RL outcomes routinely violate it. Stratified bootstrap confidence intervals — resample runs with replacement, within each task, recompute the aggregate, and take the empirical interval — make no distributional assumption and behave sensibly with a handful of runs per task. On a learning curve, shade that interval over training steps rather than drawing a single line. ## Why 'best of five' is a claim about luck A paper that plots the best of five seeds is reporting the maximum of five draws from an unknown distribution. The maximum is a biased estimator of what you would get on a fresh run: it is optimistic by construction, and the bias grows with the number of runs you selected from. Worse, it is not comparable across methods unless both were selected from the same number of runs, which is usually not stated. The reader cannot recover the quantity they care about — what happens when *they* run it once. ## The same problem inside a hyperparameter sweep This is where the mistake usually hides. Suppose you sweep a grid, one run per cell, and take the best cell. If the seed-to-seed spread *inside* a single cell is larger than the gap between the best and second-best cells, you have mostly ranked noise, and the winning cell's score is an over-estimate for the same reason a best-of-five curve is. The winner will regress when you re-run it. The fix is to re-evaluate the top few cells with a fresh set of seeds, and to report the winner's score from those fresh runs rather than from the sweep that selected it. It also reframes the sweep result: if the best cell's advantage does not exceed the within-cell spread, the honest conclusion is that the hyperparameter did not matter over the range you searched. ## Evaluation protocol details that change the number State them, because readers cannot compare otherwise. How many evaluation episodes per seed, and were they run with the stochastic policy or a deterministic one. Which checkpoint — the final one, or the best checkpoint by evaluation score, which reintroduces exactly the selection bias described above. Whether evaluation episodes used environment seeds held out from training. And, for on-policy methods, whether the reported curve is training-time episode return, which is measured under exploration noise and is not the same quantity as a clean evaluation. ## Comparing two methods When the question is 'is A better than B', the useful statements are the interval around the difference of aggregates, and the probability that a random run of A beats a random run of B — a rank-based quantity that needs no normality assumption and is meaningful even when both distributions are bimodal. A p-value from a t-test on six runs assumes a shape the data does not have; prefer the bootstrap and the rank statistic, and say plainly when the seeds cannot separate the methods.