Why Standard Machine Learning Fails on Market Data

Take a practitioner who is genuinely good at machine learning, hand them price history, and the usual result is a model with excellent validation metrics that loses money in production. This is not a skills problem and it is not bad luck.

The standard workflow, split the data, cross-validate, tune on the validation score, ship the best model, rests on assumptions that are approximately true for images and text and are flatly false for markets.

Key idea: Financial data breaks three separate assumptions at once: that samples are independent, that the data-generating process is stable, and that the environment does not respond to your predictions. Each has its own failure mode, each is invisible in a validation score, and each has a specific remedy. This course is those remedies.

In most applied machine learning, a model that is right 53 percent of the time is useless. In trading it can be a business.

That inversion has a consequence people miss. When the true edge is a couple of percentage points above chance, it sits well inside the range that noise, leakage and selection can manufacture on their own. In image classification, a leak that inflates accuracy by three points is a rounding error against a 97 percent baseline. Here it is larger than everything you were looking for.

Key idea: The lower the true signal, the higher the standard of evidence has to be, because the size of the effect you are hunting is comparable to the size of the mistakes you might be making. Most of the machinery in this course exists to keep methodological artefacts smaller than the edge, and in finance that is a demanding requirement rather than a formality.

Cross-validation assumes that holding out a sample tells you something about unseen data. That requires the held-out sample to be independent of the training ones, and financial samples are entangled in three distinct ways.

• Overlapping labels. Label a sample by the return over the next 10 days, sample daily, and consecutive labels share 9 of their 10 returns. Two rows in your dataset can be built from almost the same future.
• Serial correlation in features. A trailing 20-day volatility on Tuesday and on Wednesday are nearly the same number. Neighbouring rows are near-duplicates.
• Cross-sectional correlation. On any given day, most equities move together. A thousand names is nowhere near a thousand independent observations.

Definition: Two samples are concurrent when their label windows overlap in time, so their outcomes are determined partly by the same price moves. Concurrency is the specific mechanism that lets information cross a train-test boundary without anything obviously wrong having been done.

That experiment is reproducible in about forty lines, and it is worth running yourself, because the number is the whole argument. A validation procedure that reports skill on noise cannot be used to detect skill in anything else.

A cat in 2015 and a cat in 2025 are the same object. A market is not. Regulation changes, participants enter and leave, tick sizes change, volatility regimes shift, and the relationships a model learned can weaken or invert.

This is not drift in the usual monitoring sense, where the input distribution wanders while the underlying mapping holds. Here the mapping itself is a moving target, and a model trained across a regime boundary is fitting an average of two different worlds.

Gotcha: More history is not automatically better. Extending a training set back through a structurally different period adds rows and subtracts relevance, and the validation score will usually improve anyway because there are more samples to interpolate between. Sample size and sample relevance pull in opposite directions here, which is unusual and easy to get backwards.

The practical consequences, detection and retraining policy, get a lesson of their own later in this path.

No other machine learning domain has this property. Handwritten digits do not rearrange themselves once you learn to read them. Markets do, for two mechanical reasons.

• Your own trading moves the thing you predicted. A signal that says buy causes buying, which moves the price toward the prediction and away from the profit. Edge and size are therefore not separable, and a strategy that works at one million can be worthless at a hundred.
• Everyone else is running the same search. Any pattern discoverable from public data is being looked for by well-resourced competitors, and its returns fall as it is found and traded. Publication accelerates this.

Key idea: In supervised learning the labels are fixed and the only question is whether you can predict them. In markets the labels are produced by a process that includes participants reacting to predictions, including yours. That makes the target reflexive, and it is why a discovered edge has a shelf life rather than a value.

The fourth row is deliberately handed off. Capacity, impact and alpha decay are economic questions rather than data-preparation ones, and the catalogue's execution and backtesting courses cover them properly.

In practice: The ordering matters. Fixing validation last is the common mistake, because it means every earlier decision was made against a metric that was lying to you.

Before any modelling, a financial dataset has to survive a set of bookkeeping problems that silently insert the future into the past.

• Restatements. Company fundamentals are revised after publication. A database holding the final revised figure lets a model see an earnings number months before anyone had it. The fix is point-in-time data, which stores what was known on each date rather than what turned out to be true.
• Survivorship. An index constituent list downloaded today contains the firms that survived. Backtesting on it quietly removes every bankruptcy from history.
• Timestamp semantics. Is that field stamped when the event happened, when it was published, or when your vendor ingested it? The gap between the three is where look-ahead lives.
• Corporate actions. Splits and dividends must be adjusted consistently, and an inconsistent adjustment creates a fake return on the action date.

Gotcha: None of these is detectable from the model side. Every one produces a clean, well-behaved dataset that trains without complaint and validates well, and the only defence is auditing how each field was constructed.

The instinct when results disappoint is to reach for a stronger architecture. In this setting that instinct makes things worse, and it is worth seeing why.

A more expressive model fits whatever structure is present, and in a leaked dataset the leakage is structure. Capacity that would find a real pattern will find the artificial one faster and more completely, so upgrading from logistic regression to a deep network typically raises the validated score and lowers the live performance.

Key idea: In low signal-to-noise problems, model capacity amplifies whatever your data preparation got wrong. That is why this course spends four lessons on labels, features and validation and none on architecture: the binding constraint is almost never the model.

The corollary is a useful test. If a change to your data pipeline moves the validated score more than a change of model class does, the pipeline is what you are actually measuring.

The three lessons that follow build the dataset, in the order the decisions have to be made.

1. Labels. What are you predicting? Fixed-horizon returns are the default and are close to unusable, because they ignore risk, path and holding period. The triple-barrier method and meta-labelling replace them.
2. Features. How do you make a series stationary without deleting the memory that carried the signal? Fractional differentiation is the answer, and it is a genuine dial rather than a binary choice.
3. Validation. How do you measure anything, once you accept that neighbouring samples share futures? Purging, embargo, sample uniqueness weights, and the combinatorial variant.

In practice: This course stops at the point where you have an honest dataset and an honest score. What to do with them, backtest protocol, capacity, execution and risk, is covered by the existing courses on backtesting, trade execution and trading risk, and none of it is repeated here. The intended reading order is this course first, since those courses assume the data underneath them is sound.