Out-of-sample evidence and multiple testing

Out of sample describes an information boundary

An observation is out of sample for a fitted model when it did not influence that fit, but evaluation independence requires more than exclusion from coefficient estimation. Feature definitions, benchmark choice, tuning and rejection rules can also absorb information from evaluation outcomes. A test period that repeatedly guides revisions becomes part of development. Its dates do not restore its independence. A documented evaluation distinguishes training, model selection and final assessment, with each transformation tied to the information available at the relevant time. The important boundary concerns how observations affected decisions, not whether a file or chart happens to carry an out-of-sample label.

Chronology does not eliminate dependence

For a forward forecasting claim, training on later observations and evaluating earlier ones does not reproduce the claimed information flow. Scikit-learn's TimeSeriesSplit provides ordered splits and an optional gap between training and testing. Such a gap is a mechanical exclusion rule, not an automatic solution to every leakage problem. Overlapping target horizons can carry information across a boundary, and release delays can extend beyond a row's nominal date. Even clean chronological test observations can remain serially dependent. Consequently, the number of rows, trades or folds is not automatically the number of independent units available for estimating uncertainty.

Selection changes the interpretation of a winner

A reported result selected from many variants has benefited from more opportunities for an unusually favorable outcome. Trials include changes in features, windows, universes, benchmarks and cost assumptions, not only formally named models. Bailey and coauthors analyze backtest overfitting as a property of a selection process and show why the search history matters. Their framework is distinct from merely computing a descriptive return statistic. A positive result on an honestly untouched final sample remains evidence, but repeated searching on that same sample weakens its interpretation. Undocumented earlier trials make the magnitude of selection effects harder to assess, not safely negligible.

An original hypothetical family of null tests

The table assumes every null hypothesis is true, each test has an exact false-positive probability of 0.05 and tests are independent. For m tests, the probability of at least one false positive is 1 minus 0.95 raised to m. It is 0.336580 for eight tests, 0.641514 for twenty and 0.953930 for sixty. The expected false-positive count is 0.05 times m; this expectation does not itself require independence. These are original synthetic probability calculations, not simulated strategy results. They do not estimate the probability that any particular published result is false, which would require information beyond this family-wise calculation.

Error control requires a declared family

NIST presents the Bonferroni inequality underlying simultaneous error control. For a fixed family of m tests with valid individual p-values, a per-test threshold of 0.05 divided by m bounds the chance of any false rejection by 0.05, without requiring test independence. The table reports these thresholds for the hypothetical families. This bound can be conservative when tests are strongly related. False discovery rate control targets a different quantity: the expected proportion of false rejections among all rejections, with a convention for no rejections. Neither error rate is a posterior probability that a particular strategy is genuine or an assurance of economic usefulness.

A final test and a research diagnostic differ

Walk-forward evaluation assesses a sequence of fits under a specified updating rule; it does not necessarily assess one permanently frozen model. Nested evaluation separates inner selection from outer scoring, but its outer results can still become development information if used repeatedly for redesign. Backtest-overfitting diagnostics likewise depend on the candidate set and resampling assumptions they receive. They cannot recover omitted trials or create unseen market history. A basic sample-statistics engine does not implement these procedures by returning a mean, information ratio or drawdown. Discussion of the procedures is therefore an interpretation boundary, not a promise of automated selection correction or certification.

The remaining uncertainty is part of the result

An evaluation record identifies the selection history, untouched period, baseline, scoring rule and dependence assumptions alongside any estimate. It also distinguishes statistical significance from effect size after costs and from practical feasibility. Failure to reject a null does not prove the absence of an economically relevant effect; rejection does not guarantee persistence under a changed environment. An honest final sample can still be short or unusually favorable. The synthetic family calculation establishes only how repeated opportunities alter false-positive risk under stated assumptions. It supplies no universal performance hurdle, no validated strategy and no numerical shortcut around genuinely independent evidence.

Primary references

These sources support methods and conventions. The example values are original constructions, not observations extracted from these documents.

• Bailey, Borwein, Lopez de Prado and Zhu: The Probability of Backtest Overfitting, author-hosted February 2015 manuscript
• Scikit-learn documentation: TimeSeriesSplit
• NIST e-Handbook: Bonferroni's method

Related investigations