The Look-Ahead Problem: Your Model Already Knows

The lesson The Biases That Break It Before Statistics covers the mechanical ways a backtest lies: survivorship bias, look-ahead in the data, and the rest. All of them are properties of the dataset, and all are fixable by assembling it more carefully.

A language model introduces a different one, and the difference is that the contamination is not in your data. It is inside the model.

A model pretrained on a large corpus of internet text has almost certainly read the news from your backtest period, the analyses written afterwards, and commentary describing what those events turned out to mean. When you ask it in 2026 to assess a headline from 2019, you are asking something that has read the following seven years.

So the usual defences do not apply. Point-in-time data does not help, because the leakage is not in the data. Careful sample construction does not help, because the sample is fine. Walk-forward testing does not help, because the model does not walk forward.

This is not a hypothetical concern raised by sceptics. It has been measured, and the measurements are uncomfortable.

Paul Glasserman and Caden Lin separated the problem into two mechanisms in "Assessing Look-Ahead Bias in Stock Return Predictions Generated By GPT Sentiment Analysis" (2023).

Look-ahead bias is the obvious one: the model may have specific knowledge of the stock returns that followed a given news article. Asked whether a headline is good news, it can answer from having read what happened rather than from the headline.

The distraction effect is subtler and does not require knowledge of the future at all. General knowledge about the companies named in a text interferes with the measurement of that text's sentiment. A model asked to score a neutral headline about a company it associates with a decade of strong performance is not scoring the headline alone.

The second is the more insidious of the two, because it corrupts the measurement even where no leakage is present. It means a sentiment score is a mixture of the document and the model's priors about the entities in it, in unknown proportion.

Their proposed mitigation addresses both: anonymisation, replacing identifying entities so the model scores the text rather than the company. They note it is useful for de-biased backtesting and in out-of-sample implementation.

That is a real technique with a real cost, since anonymising a document removes information a human analyst would legitimately use.

The instinctive fix is to instruct the model to ignore anything after the date in question. Suproteem Sarkar and Keyon Vafa tested whether that works, in "Lookahead Bias in Pretrained Language Models" (2024).

Their method builds direct tests for the bias, resting on the observation that some events are unpredictable given a prespecified information set. If a model's answers about a past date reflect something genuinely unforeseeable at that date, the knowledge came from pretraining rather than from reasoning.

They found the bias in applications including predicting corporate risk factors from earnings calls, and their clearest illustration is this: models use their knowledge about COVID-19 when predicting risk factors of companies before the pandemic, even when instructed not to use any information about future events.

Read that carefully, because it settles a common assumption. The instruction was given. The model complied in form and not in substance. Pretraining knowledge is not a set of facts the model can be asked to set aside; it is diffused through the weights that produce every token.

The authors discuss the limitations of prompting-based approaches directly, and their conclusion points elsewhere: the issue is addressed by using models whose pretraining data contains only language produced prior to the analysis period.

That is a much stronger requirement than a well-written prompt.

Every arrow converges on the same box, and that is the point of the diagram.

The standard toolkit for honest backtesting was built for contamination in datasets. It is a good toolkit and it is aimed at the wrong place. The model is a fixed artefact that read the future once, at training time, and no amount of care in constructing the evaluation changes what it read.

This reframes what a model's knowledge cutoff is for. It is usually treated as a limitation, a date after which the model is ignorant. For this purpose it is the opposite: it is the only clean boundary available, and it is the single most important fact about a model you intend to evaluate historically.

Everything before the cutoff is potentially contaminated. Everything after it is potentially clean.

That is a severe constraint, because it means a model released this year can only be honestly evaluated on data from this year, which is a short sample with few independent events, and the deflated Sharpe ratio from Selection Bias and the Deflated Sharpe Ratio applies with full force to whatever you find in it.

A natural hope is that contamination adds noise, which would attenuate a measured result rather than manufacture one. It does the opposite, and the reason follows from what the model absorbed.

The corpus does not contain a neutral record of what happened. It contains commentary written afterwards by people who knew the outcome. Analyses of why a stock fell, retrospectives on which warnings were prescient, narratives that make the sequence look inevitable.

A model trained on that has absorbed the outcome-conditioned framing, and asked about the antecedent it will tend toward the interpretation that history endorsed.

So the error is not random. It is systematically correlated with the realised outcome, which is precisely the direction that makes a backtest look profitable.

Combine that with the selection process in Selection Bias and the Deflated Sharpe Ratio and the position is worse than either problem alone. A researcher tries several prompts, several models and several scoring schemes, and keeps what backtested best. The best is now selected on a metric that is inflated by contamination, so the maximum of many draws is being taken over a distribution whose mean is already too high.

The deflated Sharpe ratio corrects for the number of trials. It does not correct for a biased estimator, and there is no established correction for this one.

Four approaches, in ascending order of how much they cost and how much they establish.

Evaluate only after the cutoff. The cleanest and most restrictive. Take a model with a known cutoff and test exclusively on data after it. This is what Lopez-Lira and Tang did with post-knowledge-cutoff headlines, and it is why their result is worth taking seriously.

The cost is sample size. A few months of post-cutoff data contains few independent market events, so confidence intervals are wide and the multiple-testing problem is severe relative to what you can learn.

Anonymise the input. Glasserman and Lin's approach: strip entity identifiers so the model scores text rather than recalling a company. This targets both the look-ahead and the distraction effect, and it degrades the semantics, since real analysis uses knowing who is speaking.

Use a temporally frozen model. Sarkar and Vafa's conclusion: a model whose pretraining data contains only language from before the analysis period. This genuinely solves it, and the cost is that such models must be built deliberately and are far behind the frontier.

Run forward. Deploy the signal on data that does not exist yet and record results as they arrive. This is not a backtest and it is the only unambiguous evidence available, at the cost of taking as long as it takes.

The fourth is the one serious operations end up relying on, which changes the research cycle from months of backtesting to a year of paper trading.

One consequence is easy to miss because it survives every mitigation above.

Suppose the evaluation was perfect and the signal is genuine. The model still gets replaced. A new version arrives with a later cutoff, different training data and different behaviour, and every score it produces differs from its predecessor's.

So the historical record was generated by an artefact that no longer exists. The strategy's track record describes a system you are not running.

That is a species of regime change with no market event behind it, and it is unusual. Ordinarily a strategy degrades because participants adapt or conditions shift; here the measurement instrument was swapped out.

Three practical consequences follow.

Pin the model version in anything whose results must be comparable over time, exactly as the containers lesson argued for pinning images by digest.

Re-validate on every upgrade, treating a new version as a new signal rather than an improvement to an existing one.

Expect no vendor guarantee of behavioural stability. Providers deprecate versions on their own schedule, so a strategy's dependency on a specific model is a dependency you do not control.

The general form: a signal computed by a third-party model has a supply chain, and its stability is somebody else's product decision.

Any published or internal claim about a language model producing trading signals should survive all seven. Most do not survive the first two.

The conclusion is not that the technology is useless in this domain. It is that the evidentiary bar is higher here than anywhere else, because the standard method of establishing that a signal works is structurally compromised in a way that inflates rather than attenuates.

That reframes what the first lesson called the strong applications. Extraction is defensible partly because its evaluation is not contaminated: whether a model correctly pulled a guidance range from a filing is checkable against the filing, and no knowledge of subsequent returns helps.

The next lesson applies this checklist to the most prominent published result in the field, which passes the first two questions and still says something different from what it is usually reported as saying.