Selection Bias and the Deflated Sharpe Ratio

Suppose you test one hundred strategies. Every single one is pure noise, with a true expected return of exactly zero.

Each produces a backtested Sharpe ratio that is a random draw scattered around zero. Some come out negative, some positive, and one comes out highest.

You keep that one. Its Sharpe looks respectable.

Nothing went wrong here. No bug, no leaked data, no carelessness. The selected strategy has a good Sharpe because you selected on Sharpe, and the maximum of a hundred draws from a zero-mean distribution is reliably well above zero.

This is the entire mechanism of backtest overfitting, and it is worth sitting with how ordinary it is. The researcher did not overfit through negligence. They did what research looks like: generate candidates, evaluate them, keep the best.

That procedure, applied to a fixed sample, manufactures impressive results from nothing. And the more thorough the researcher, the more variants they try, and the more inflated the winner.

Diligence makes it worse. That is the uncomfortable part, and it is why the fix has to be statistical rather than a matter of working harder.

The effect is quantifiable, and the numbers are worth internalising because intuition badly underestimates them.

For N independent draws from a standard normal distribution, the expected maximum grows roughly like the square root of twice the log of N.

trials N     expected max of N standard normal draws
      10        ~1.54
     100        ~2.51
   1,000        ~3.24
  10,000        ~3.85
 100,000        ~4.42

Two readings matter.

The growth is slow, only logarithmic, which sounds reassuring. Going from a thousand trials to a hundred thousand moves the expected maximum by about 1.2.

But the level is high even for modest N. At a hundred trials, pure noise is expected to produce a best result around 2.5 standard errors above zero. Under conventional significance testing, that clears the bar comfortably. It would be called a discovery.

And a hundred trials is a small search. A researcher sweeping three parameters over ten values each has already run a thousand. Nobody thinks of that as multiple testing; it is called tuning.

The correction therefore depends on how many things you tried, which is a number most research processes never record.

David Bailey and Marcos Lopez de Prado proposed the standard correction in "The Deflated Sharpe Ratio: Correcting for Selection Bias, Backtest Overfitting and Non-Normality" (2014).

The idea follows directly from the previous step. If you know roughly how many trials were run, you know what the best Sharpe would look like under the null hypothesis that none of them works. The observed Sharpe must beat that benchmark, not zero.

The deflated Sharpe ratio gives the probability that the observed value exceeds what selection alone would produce. It corrects for four things together:

Number of trials, the selection effect just described. Sample length, since a short backtest estimates Sharpe imprecisely and so produces wider extremes. Skewness, because negative skew, the pattern of many small gains and rare large losses, makes a given Sharpe less impressive. And kurtosis, since fat tails widen the sampling distribution.

The last two matter because the classic Sharpe assumes normally distributed returns, and trading strategies routinely are not. Strategies that sell insurance-like risk look excellent by Sharpe precisely because their losses are rare.

The practical consequence: a Sharpe of 1.5 from a search of ten thousand variants can deflate to statistically indistinguishable from zero.

The same group developed a complementary diagnostic that asks a different question.

Bailey, Jonathan Borwein, Lopez de Prado, and Qiji Jim Zhu introduced the probability of backtest overfitting (PBO) in the Journal of Computational Finance (2017), alongside their earlier and pointedly titled "Pseudo-mathematics and Financial Charlatanism: The Effects of Backtest Overfitting on Out-of-Sample Performance" in the Notices of the American Mathematical Society (2014).

PBO estimates the probability that the strategy you selected as best in-sample will underperform the median of your candidate set out-of-sample.

The procedure is a combinatorial resampling. Split your return series into many blocks. For every way of dividing those blocks into an in-sample and an out-of-sample half, find the best strategy in-sample and record where it ranks out-of-sample. Do this across all the splits and count how often the in-sample winner lands in the bottom half afterwards.

A high PBO means your selection procedure is picking noise, and the diagnosis is about the process, not any individual strategy.

This is what makes it valuable: it evaluates your research methodology. A team with a high PBO has a broken workflow, regardless of what any single backtest shows.

The selected strategy's backtested performance decomposes into a real component and a selection component. Out of sample only the real part survives, so the drop equals whatever share was selection, which is why more search means more disappointment.

This is not only a practitioner failing. Published financial research has the identical structure, and the most cited treatment is worth knowing.

Campbell Harvey, Yan Liu, and Heqing Zhu published "...and the Cross-Section of Expected Returns" in the Review of Financial Studies, volume 29, issue 1, in January 2016.

Their observation: hundreds of published papers propose hundreds of factors claiming to explain stock returns. All of them are tested against essentially the same market history. That is a multiple testing problem of enormous scale, conducted collectively across a profession, and no individual author can see it.

Their conclusion is blunt. Given the extent of this data mining, it makes neither economic nor statistical sense to judge a newly proposed factor by the usual threshold of a t-statistic above 2.0. A new factor should clear a much higher hurdle, a t-statistic greater than 3.0.

Their framework also accounts for correlation among the tests and for publication bias, since results that fail never appear and so are invisible to any later count.

The implication for practitioners is direct. If the published literature needs t above 3.0 after peer review, an internal backtest from an unrecorded search deserves at least as much scepticism.

Every correction here needs one input: how many trials did you run? Getting that number right is harder than it sounds, and understating it defeats the whole exercise.

The count is not the number of backtests you saved. It includes:

Every parameter combination considered, including ones abandoned after a glance. Every variant explored and discarded, including the ideas that did not survive an afternoon. Everything your colleagues tried on the same data, because the history is shared even when the researchers are not. Everything tried in previous years on that dataset. And every idea you read about, since the literature is itself a filtered set of survivors.

That last category is the one that makes strict accounting impossible. You cannot count trials you never saw.

So treat the trial count as a lower bound you should deliberately inflate. If you think you ran two hundred, use two thousand.

The cheapest habit that helps is logging every backtest automatically, including the failures. Most teams delete those, which destroys precisely the information needed to interpret the survivor. A researcher who cannot say how many variants preceded a result has not produced an interpretable result.

How to read a backtested Sharpe once selection is taken seriously.

The last row carries more weight than any statistic. A strategy with a reason to work, an identifiable participant on the other side with a motive to trade against you, is a different object from a pattern found by search. The mechanism is a hypothesis formed before the data, which is what the corrections above are trying to approximate.

Two gotchas to close on.

Deflation needs an honest N, and N is unknowable. These corrections improve your reasoning; they do not certify a result. Treat a deflated Sharpe as a sanity check, not a licence.

Passing statistically is not passing. You have established the pattern is unlikely to be pure chance. Whether it survives trading costs and your own market impact is untouched by any of this, and is the subject of the final lesson.