A trading rule's backtest Sharpe is 2.5 in-sample but 0.3 the following year. What happened?

## In-sample risk versus out-of-sample risk The optimism of training error is usually taught with squared error and a fitted regression, which hides how general it is. It applies to **any performance number computed with the same data that shaped the thing being scored**, and to any metric: accuracy, AUC, lift, revenue per user, or a Sharpe ratio. The two quantities in play here are: - **In-sample risk**: how the rule performed over the history used to construct it. - **Out-of-sample risk**: how it performs on data that took no part in its construction — the only number that forecasts the future. A backtest Sharpe of 2.5 that becomes 0.3 is not a mystery or a betrayal; it is the expected behaviour of an in-sample statistic reported as if it were out-of-sample. ## Where the degrees of freedom came from The rule may have only two visible parameters — say a lookback window and an entry threshold. That is a misleading count. Everything that was tried and discarded on the same history spends freedom: - every value of the lookback that was swept, - every threshold, stop-loss and holding period tested, - every choice of universe, sector filter or de-listing rule, - every earlier idea abandoned because its backtest looked poor, - every re-run after "fixing" the data in a way that improved the curve. Each of these is an evaluation of a noisy statistic on one finite history. If you evaluate 300 variants, and the true edge of all of them is zero, the *best* of the 300 will still show a healthy positive Sharpe purely from sampling variation — and it is the best one you will report, because that is how you chose it. Selection is a fitting procedure. The reported number is a maximum, and the expectation of a maximum sits well above the expectation of any single draw. This is the same mechanism as adding free parameters to a regression: more freedom exercised on one dataset, more of that dataset's noise absorbed into the answer, larger gap between the measured number and the truth. ## The give-away symptoms When you inherit a result like this, look for the fingerprints of an in-sample number: - **Nobody can state how many variants were evaluated.** If the search is uncounted, the optimism is unbounded. - **The rule is fragile to small perturbations.** Shifting the entry threshold or the window by one step collapses the result — real edges are usually a plateau, fitted noise is usually a spike. - **Performance is concentrated in a few periods.** A handful of episodes carrying the whole result is what a noise-fitted rule looks like. - **The final specification includes clauses whose only justification is the backtest** — an excluded month, a filtered sector, a rounded threshold. ## What to do instead The fix is conceptual before it is procedural: *decide which data is allowed to influence the rule, and never score on that data*. In practice that means holding back a stretch of history that plays no part in construction, freezing the specification, and only then measuring — and if you go back and change the rule after seeing that number, it has become in-sample too and is spent. Forward performance after freezing is the strongest evidence, because it cannot have been searched over at all. It also means budgeting the search. If you know you will try many variants, expect the winner's score to be inflated and discount it before making any decision. The honest headline is not "this rule earns 2.5" but "the best of 300 variants earned 2.5 on the history that selected it, which is consistent with no edge at all". ## A second, separate explanation to acknowledge A real regime change in the following year is a distinct cause with a distinct signature, and a good answer mentions it without hiding behind it: if the world genuinely changed, other unrelated strategies degrade at the same time and the rule's assumptions can be named and checked. In practice the search-driven explanation is far more common, and "the market changed" is the standard way a fitted-noise result avoids a post-mortem. Rule out the optimism first, because it is the explanation you can quantify by counting what was tried. ## Interview register Do not lead with the market. Lead with the count of things tried on the same history, connect it to the optimism of an in-sample statistic, note that the effect applies to Sharpe exactly as it applies to squared error, and finish with the discipline you would impose: a slice of data that never touched construction, a frozen specification, and a stated expectation of shrinkage before anyone sees the number.