What a Backtest Actually Claims

In quantitative trading, almost every decision traces back to a single object: the backtest. It decides which strategies get capital, which researchers get hired, and which funds get launched.

It is also, reliably, the most misleading artifact in the field.

The pattern is well documented and repeats across firms. A strategy shows a beautiful equity curve in simulation, goes live, and returns something between a fraction of the backtested performance and nothing at all. Not occasionally, but as the default outcome.

The interesting question is why this keeps happening to intelligent, quantitatively trained people who are not trying to deceive anyone.

The answer is that a backtest looks like a measurement and is not one. It looks like you are reading a fact off the historical record. You are not. You are asserting a counterfactual, and counterfactual claims need assumptions the data cannot supply.

This path is about being precise on exactly what a backtest claims, which of those claims break, and what a defensible research protocol looks like.

Write out what a backtest asserts, in full, because the vagueness is where the trouble hides.

Had I been running this exact strategy, with this exact capital, over this exact period, using only information available to me at each moment, and trading at prices I could actually have obtained, my account would have followed this path.

Every clause is a separate assumption, and every one can fail.

Had I been running this exact strategy assumes the rules were fixed in advance, not discovered by looking at the data.

With this exact capital assumes your size did not move the market.

Using only information available at each moment assumes no future data leaked into any decision.

At prices I could actually have obtained assumes your orders would have filled, at those prices, in that size.

The last clause is the giveaway: my account would have followed this path. That is a statement about a world that did not occur. It is a counterfactual, in the strict sense, and no amount of historical data makes it a direct observation.

Framing a backtest as a counterfactual puts it in familiar territory: it is an interventional question, not an observational one.

You are not asking what the market did. That is observation, and the record answers it. You are asking what would have happened had you acted, which is a question about intervening in a system you were not part of.

The distinction matters because it tells you which failures to expect.

An observational claim fails through measurement error. A causal claim fails through confounding and unstated assumptions, and those failures do not shrink as you collect more data.

There is a second consequence, easy to miss. In an ordinary causal question, the system does not know you are asking. Markets do. If you had traded, prices would have moved, other participants would have responded, and the history you are testing against would itself have been different.

So a backtest assumes not only that you could have acted, but that the world would have carried on unchanged while you did. For small size that approximation is fine. For large size it is false, and the fourth lesson returns to it.

Here is the structural problem that makes this harder than ordinary science.

In a laboratory you can run an experiment again. Draw a fresh sample, replicate, and let independent evidence accumulate.

Financial markets give you one realised history. There is exactly one path that prices actually took. You cannot re-run 2015 under different conditions and see whether your strategy still works.

So every strategy you test is evaluated against the same data. And this is the asymmetry that drives everything in the third lesson: strategies are cheap and history is fixed.

A researcher can generate thousands of variants in an afternoon. Change a lookback window, add a filter, adjust a threshold, swap a universe. Each is tested against the identical, finite history.

With enough variants, some will fit that history beautifully for no reason other than chance. Not because the researcher was careless, but because searching a large space against a fixed sample is guaranteed to turn up flattering coincidences.

The backtest cannot distinguish those coincidences from real structure. Both produce the same beautiful curve.

Strategies are usually compared on one number, the Sharpe ratio: excess return divided by volatility, annualised.

Sharpe = (mean return - risk free rate) / standard deviation of returns
         scaled by sqrt(periods per year)

It answers a reasonable question: how much return per unit of risk? A Sharpe of 2 in a backtest looks strong; 0.5 looks marginal.

Three properties are worth knowing before you trust one.

It is an estimate, with error. A Sharpe computed from two years of daily data has a wide confidence interval. Reporting 1.8 without an interval implies precision the sample does not support, and short backtests produce the widest intervals and the most confident-looking numbers.

It assumes volatility captures risk. Standard deviation treats an upside surprise like a downside one and says nothing about tails. A strategy that earns steadily then loses catastrophically can show an excellent Sharpe right up until it does not.

It is the number being selected on. If you searched many strategies and kept the best Sharpe, the value you are looking at is the maximum of many draws, not a typical one. That single fact is the subject of the third lesson.

The failures are not variations of one problem. They are separate, they compound, and each needs its own defence, which is why the rest of this path treats them one at a time.

The standard defence is to split the data. Develop on one period, evaluate on another that the strategy has never seen.

The logic is sound. An overfitted strategy has memorised noise specific to the development period, and that noise is not present in the held-out data, so performance should collapse there.

In practice the defence erodes, for a reason that has nothing to do with discipline.

You look at the out-of-sample result. It disappoints. You adjust the strategy and look again.

At that moment the held-out period has become part of your development set. Not formally, but informationally: it is now influencing your choices. Repeat a few times and the out-of-sample test is measuring how well you fit the out-of-sample data.

This is the same mechanism as the garden of forking paths in empirical research, arriving in a setting where the temptation is stronger, because the researcher genuinely wants the strategy to work and each individual adjustment is defensible.

The practical implication is uncomfortable: a held-out set is a consumable resource. Each look spends some of it, and nothing about the code enforces the budget.

Being precise about the instrument's range prevents most of the damage.

The second row deserves emphasis, because it is the use most people undervalue. A backtest is an excellent debugging tool. It surfaces logic errors, sign flips, and unhandled edge cases. That value is real and does not depend on any of the statistical claims.

The honest framing of the whole exercise: a backtest is a necessary filter, not evidence of an edge. Failing one is strong evidence against a strategy. Passing one is weak evidence for it, and how weak depends entirely on how many strategies you tested to find it.

The next lesson covers the failures that make a backtest wrong before any statistics are involved.