Why Reported Order Book Results Do Not Replicate

Order book prediction has a reproducibility problem with a recognisable shape. A paper reports high accuracy on a benchmark. A practitioner reimplements it, gets a similar number, tries it on their own data, and it collapses. Nobody was dishonest and the code often runs correctly.

Four mechanisms account for most of this, and they compound rather than merely adding.

1. The label is mostly nothing, so accuracy measures the wrong thing.
2. The evaluation leaks, in a way that rewards capacity.
3. The horizon is chosen after seeing results.
4. Accuracy and profitability are only loosely related.

Key idea: None of these is about the model. Each is a property of how the problem was set up, which means each is checkable before training and none is fixable afterwards.

The standard target is the direction of the mid-price change over the next $k$ events. That target has a structural property people rarely quantify: at short horizons, the mid usually does not move at all.

Prices live on a discrete tick grid, and the mid sits between the best bid and ask. If the spread is one tick and neither side changes, the mid is exactly unchanged, and at high frequency most events, an order added deep in the book, a cancellation, a partial fill, do not move the touch.

The problem eases as the horizon lengthens, and it does so in a way worth seeing, because it explains why horizon choice is such a powerful lever.

The two lines coincide while flat is the majority class and separate once up and down each overtake it, past about 20 events, after which the baseline is set by the larger directional class instead.

Gotcha: A model reporting 70 percent at $k=5$ and one reporting 45 percent at $k=100$ cannot be compared. The first is exactly the no-information baseline; the second is a few points above it. The headline number moves with the horizon far more than it moves with model quality, so any accuracy quoted without its horizon and its majority-class baseline is uninterpretable.

Because pure direction is so imbalanced, most work uses a threshold: label up if the mid rises by more than $\alpha$, down if it falls by more than $\alpha$, flat otherwise. That threshold is a free parameter with an enormous effect.

Gotcha: The threshold and the horizon together give two knobs that move reported accuracy over a very wide range, and both are usually chosen by looking at results. That is selection on the test set, and it is why a paper's headline number can be reproduced on its own benchmark and nowhere else: the configuration was fitted to that dataset even though no model parameter was.

The defence is to fix the horizon and threshold from the economics, what move is large enough to trade after costs, before looking at any accuracy figure at all.

There is a subtler problem with mid-price direction as a target. Part of its movement is mechanical rather than informational.

When the spread is one tick, the mid alternates as the touch flickers between states, and that flicker is partly a consequence of the discrete grid rather than of any change in what the asset is worth. A model can learn to anticipate it, because book state genuinely predicts which side moves next, and that prediction is real without being tradeable.

Key idea: You cannot capture a half-tick mid move. To profit you must cross the spread, paying a full tick, or rest passively and be filled, which means waiting and accepting adverse selection. A model that predicts the mid perfectly at one-tick resolution has predicted a quantity nobody can transact at.

This is the strongest argument for using the microprice, or better, for labelling by whether a specific tradeable action would have profited, which is what the fourth lesson turns to.

The general problem, that overlapping labels break cross-validation, is established in the catalogue's Financial Machine Learning course. Order book data adds mechanisms of its own.

• Overlapping input windows. Consecutive samples share almost their entire history window, so a random split places near-identical inputs on both sides.
• Overlapping labels. With a $k$-event horizon, consecutive labels share $k-1$ events, exactly as in the general case.
• Normalisation across the split. Standardising features using statistics computed over the whole dataset, an extremely common convenience, leaks the test distribution into training.
• Session boundaries. Sampling within a day and splitting randomly means a model sees the afternoon while being tested on the morning.

Gotcha: The third is the most frequent and least noticed, because normalisation feels like preprocessing rather than modelling. Fitting a scaler on the full series before splitting is a single line and it contaminates every result computed afterwards, including the baseline you were comparing against, which hides the problem by inflating both numbers.

The fix is the same in all four cases: contiguous splits, purge by the label horizon, embargo by the input window length, and fit every transform inside the fold.

Public benchmarks are essential for comparability and they have a specific failure mode as they mature.

A fixed dataset used by many papers over many years accumulates collective selection. Each author tunes against it, keeps what works, and publishes. Nobody overfits it individually, and the literature as a whole does, because the choices that survived were the ones that happened to suit that data.

Key idea: The trial count that governs a benchmark's reliability is the total across everyone who has ever used it, and that number is unknowable and large. This is why a result reproducing on the benchmark and failing on fresh data is the expected outcome rather than a surprise, and why the only convincing evidence is performance on data the field has not seen.

Market data has a compounding version of this: the market itself changes, so an older benchmark also describes a market that no longer exists.

Even with identical methodology, the same model applied to different data gives very different results, and the axes of variation are rarely stated.

In practice: A model validated on large-cap equities in one market can fail entirely on small caps in the same market, on the same names in a different tick regime, or on the same names five years later. None of that is a modelling failure; it is the result being narrower than its statement.

The honest form of a claim here always names the instrument class, the venue, the period and the tick regime, because a result without them is not reproducible even in principle.

Taken together the mechanisms multiply rather than add, and the order in which they apply matters.

1. The baseline is high, so a headline accuracy that sounds impressive may be at or below no-information performance.
2. Horizon and threshold are tunable, so the configuration can be selected to make a number look good on this dataset.
3. Leakage inflates everything, and inflates high-capacity models more, which flatters exactly the comparison being reported.
4. The benchmark has been collectively fitted, so surviving choices suit that data specifically.

Key idea: A result that passes all four filters is genuinely informative, and the four filters are cheap to apply. Report accuracy alongside the majority-class baseline. Fix horizon and threshold from economics before measurement. Use purged contiguous splits with transforms fitted inside the fold. Validate on data the benchmark never contained.

None of that requires a better model, and all of it can be done before training. The final lesson turns to what should be measured instead of accuracy, which is where the remaining gap between a good classifier and a working strategy lives.

Applied to your own work, or to a paper you are deciding whether to believe.

1. Is the majority-class baseline reported? If not, the accuracy figure cannot be interpreted at all.
2. Are the horizon and threshold stated, and were they fixed in advance? Both move results more than the model does.
3. Is the split contiguous and purged, with the embargo sized by the input window? A random split on overlapping windows invalidates everything downstream.
4. Were scalers and any other transforms fitted inside the fold?
5. Is there a features-plus-trees baseline on identical inputs? Without it, depth has not been shown to contribute.
6. Are the instrument class, venue, period and tick regime stated?
7. Has it been validated on data outside the benchmark?
8. Is the latency of inference reported against the decay of the signal?

In practice: Items 1 and 3 alone will resolve most disagreements about whether a result is real. They cost minutes, they require no new data, and they are the two most commonly missing.