The Biases That Break It Before Statistics

The previous lesson set up the statistical problem: with enough attempts, something fits by chance. That problem is subtle and requires real machinery to address.

This lesson is about a cruder class of failure, and it is worth separating clearly. These are errors that make the simulation factually wrong as a reconstruction of the past. They are not about whether the edge is real. They are about whether the number on the screen corresponds to anything at all.

The distinction matters practically. A statistically overfitted strategy at least described the historical data correctly; it just will not persist. A mechanically broken backtest describes a history that never existed, and its number means nothing.

The good news is that these failures are, in principle, fixable. They are engineering and data problems with known solutions, unlike selection bias, which no amount of care eliminates.

The bad news is that they are easy to introduce, hard to see, and every one of them biases in the same direction: they make the strategy look better than it was.

Look-ahead bias is using information in a decision that would not have been available when the decision was made.

Stated that way it sounds like an obvious blunder nobody would commit. In practice it is the single most common defect in backtesting code, because it enters through paths that look entirely innocent.

The canonical case is the closing price. Your signal says buy when the close exceeds a moving average. But the closing price is not known until the market closes, so an order placed at that price is placed using information from the moment it becomes final. The backtest fills you at a price you could not have traded on.

The subtler cases are worse because they survive review.

Normalising over the whole sample. Scaling a feature by its mean and standard deviation computed across the entire dataset embeds future information into every early observation.

Fitting a model once on everything. Any parameter estimated using the full history is known to the strategy before it should be.

Bar timestamps. A bar labelled 09:30 usually contains data from 09:30 to 09:31. Acting on it at 09:30 is acting on the next minute's data.

The unifying test: for every value your strategy reads, ask when was this knowable?

Survivorship bias is testing on a universe that was chosen using knowledge of what survived.

The mechanism is easy to state. Take today's index constituents and backtest ten years. Every company in that list is one that did not go bankrupt, get delisted, or collapse over the decade, because if it had, it would not be in today's index.

You have not tested a strategy. You have tested a strategy on a pre-filtered list of winners, selected with information from the end of the period.

The effect is directional and can be large. Returns are inflated, and drawdowns are understated because the worst outcomes have been removed from the sample entirely. It compounds over long horizons, since the longer the window, the more failures have been filtered out.

The same defect appears wherever a dataset has been assembled with hindsight: fund databases that drop closed funds, cryptocurrency data covering only tokens still listed, or an equity file where delisted tickers are simply absent.

The fix is a point-in-time universe: the list of instruments as it stood on each historical date, including everything that later failed. Vendors sell this precisely because reconstructing it is difficult and doing without it is worse.

Survivorship is one instance of a general requirement: your data must reflect what was known then, not what is known now. Databases are usually built for the opposite purpose.

Restatements. A company reports earnings, then revises them months later. Most databases store the corrected figure against the original date. A backtest reading it trades on a number nobody had.

Reporting lag. Quarterly results describe a quarter that ended weeks earlier. Using them from the quarter-end date grants the strategy several weeks of foresight.

Index changes. Additions and removals are announced before they take effect, and databases often record only the effective date.

Corporate actions. Splits and dividends require adjusted prices, and adjustment factors are applied retroactively across the entire history.

That last one has a specific trap worth naming. A price series adjusted for a split that happened in 2020 shows different prices for 2015 than the screen showed in 2015. A strategy with a rule like "buy below 50 dollars" is therefore evaluated against prices that never traded.

The general rule: a database records the present's best understanding of the past. A backtest needs the past's understanding of itself, which is a different and much rarer product.

Every one of these leaks moves the result the same way, which is why a backtest with several small defects can look outstanding rather than merely slightly optimistic. They accumulate rather than offset.

Individually checking for each bias does not scale, because there are always more ways to leak than you can enumerate. The durable fix is architectural.

Build the backtest as an event loop that cannot see the future, rather than as vectorised operations over a full history.

# the strategy is handed data as it arrives, and can only ask
# for what has already been delivered
for timestamp, event in market_data_stream:          # strictly increasing
    portfolio.mark_to_market(event)
    orders = strategy.on_event(timestamp, event)     # sees only the past
    for order in orders:
        fill = exchange.execute(order, next_bar)     # fills at the NEXT bar
        portfolio.apply(fill)

Two properties do the work.

The strategy is handed data rather than querying a dataframe, so it is structurally incapable of reading a future row. Leakage becomes impossible rather than merely discouraged.

And orders fill at the next bar, never the one that generated the signal, which enforces the ordering that decisions precede executions.

The cost is real: event-driven backtests are far slower than vectorised ones, which is why researchers reach for the vectorised version and why it keeps producing leaks. A common compromise is to explore vectorised, then confirm any candidate in an event-driven engine before believing it.

Since leaks hide, it helps to have tests that surface them without knowing where they are.

Shuffle the labels. Replace your signal with random noise of the same shape and rerun. A correct backtest should produce roughly zero return before costs and clearly negative after them. Anything profitable means your harness is generating returns on its own.

Shift the signal forward one bar. If performance barely changes, your signal was probably already using information from that bar. A genuine edge usually degrades sharply when delayed.

Check the trade timestamps. Pull the actual fills and confirm every one is strictly after the data that triggered it. This catches off-by-one indexing, which is the most common source of the problem.

Compare against a naive benchmark. A strategy vastly outperforming buy-and-hold with a high hit rate and small drawdowns is more likely to have a leak than an edge.

The useful instinct is inverted from normal engineering. Here, an unusually good result is a bug report. Sharpe ratios above roughly 3 or 4 on liquid assets with daily data are rare in reality and common in flawed backtests, so treat them as a prompt to audit rather than a reason to celebrate.

What to verify before a backtest result deserves any statistical discussion at all.

Two things to carry forward.

These biases all point the same way. None of them makes a strategy look worse. That asymmetry is why a backtest with a handful of minor defects does not come out slightly optimistic, it comes out spectacular, and spectacle is exactly what gets a strategy funded.

Passing every check means the simulation is honest, not that the edge is real. You have established that the number describes something that actually could have happened. Whether it happened for a reason, or by coincidence among the thousands of variants you tried, is a completely separate question, and it is the subject of the next lesson.