Evaluating a Book Model Honestly

A classifier is scored on how often it is right. A strategy is scored on money. Those come apart, and in this setting they come apart severely.

Being right on a hundred moves of a tenth of a tick and wrong on ten moves of a full tick is 91 percent accuracy and a loss. Accuracy weights every prediction equally; profit weights each by the size of the move and by whether you could act on it.

Key idea: Accuracy is a proxy that happens to be convenient. It is a reasonable proxy when errors are symmetric and outcomes are similar in magnitude, and neither holds here: move sizes are heavy-tailed, and the cost of acting is comparable to the typical move. The rest of this lesson replaces it with measures tied to a decision you could actually take.

Suppose you predict direction with accuracy $p$, capture a move of typical size $m$ when right, lose $m$ when wrong, and pay $c$ in costs per round trip. Expected profit per trade is

$(2p - 1)\,m - c$

so breaking even requires

$p \geq \frac{1}{2} + \frac{c}{2m}$

A round trip that crosses a one-tick spread twice costs about one tick. If the move you are chasing is also about a tick, the ratio is 1.0 and the required accuracy is 100 percent.

Gotcha: This is why high-frequency directional prediction is so much harder than the accuracy figures suggest, and why most of the money at this frequency is made providing liquidity rather than taking it. A market maker earns the spread instead of paying it, which moves them to the left end of this curve rather than the right.

Replacing accuracy means choosing measures that weight predictions the way money does.

Key idea: The last row is the one that matters most and is reported least. A model that is unreliable overall but excellent on its most confident 1 percent of predictions is a good trading model, because you only trade the 1 percent. Overall accuracy averages the useful predictions together with the ones you would ignore, and can therefore rank a usable model below an unusable one.

So the right question is not how accurate the model is but how good it is on the subset where it is confident enough to act.

The cleanest fix is to stop predicting price moves and start predicting outcomes of specific trades. That removes the gap between metric and objective by construction.

Instead of "will the mid rise over the next 20 events", the label becomes "if I buy at the ask now and exit at the bid within 20 events, do I profit after fees". Costs, the spread and the exit rule are inside the label rather than applied afterwards.

def label_action(book, t, horizon, fee):
    entry = book.ask[t]                      # pay the spread to get in
    exits = book.bid[t+1 : t+1+horizon]      # pay it again to get out
    best  = exits.max()
    return 1 if (best - entry - 2*fee) > 0 else 0

In practice: The class balance collapses when you do this, because most moments are not profitable trades, and that is the honest picture rather than a problem to be fixed by rebalancing. It also connects directly to the triple-barrier labelling in the Financial Machine Learning course: define the trade, define the exits, and let the label record what the trade did.

The accuracy of such a model is directly interpretable, since every positive prediction is a trade you would place.

Capacity and edge are inversely related: the faster the signal decays, the less time you have to build a position, and the more your own trading moves the price against you.

Key idea: Report edge and capacity together, always. "55 percent accuracy on a two-second horizon" is not a result. "55 percent accuracy on a two-second horizon, supporting 5,000 shares per signal, in names averaging 300,000 shares a day" is one, because it can be multiplied out into a business.

This is also why capacity limits push firms toward slower signals, where the decay window is longer and larger positions can be built, at the cost of competing with a much larger pool of participants.

Backtests at this frequency routinely omit costs that dominate the result, and the omissions are systematic rather than random.

Gotcha: Every omission on that list biases the same way. Not one of them would make a strategy look worse than reality, so an incomplete cost model is not noisy, it is systematically optimistic. A backtest showing a smooth equity curve at high frequency is usually showing you which costs were left out.

The fourth row deserves particular attention. Passive fills are not random: you are filled by someone who chose to trade against you, which means your fills are concentrated in the moments when trading was worst for you.

A signal in public book data is not a durable asset, and its decay has identifiable causes.

• Competition. Others find the same pattern in the same data, and their trading removes it. The more mechanical and obvious the signal, the faster this happens.
• Your own impact. Trading on the signal moves the price toward the prediction, which is the mechanism by which the pattern is removed. Success consumes the opportunity.
• Market structure change. Tick regimes, venue rules and fee schedules change, and a pattern that depended on the old structure disappears when it does.
• Participant turnover. Patterns caused by a particular participant's behaviour end when that participant changes their system.

In practice: Treat a signal's half-life as a property to be measured rather than assumed. Track the realised edge over time and expect a decline; a strategy that shows no decay is more likely to be mismeasured than to be permanent. Budget for continuous rediscovery, because in this area maintaining an edge is an ongoing cost rather than a one-off research project.

The most useful discipline is to state, before deployment, what evidence would show the result was wrong. It converts a hope into a testable claim.

1. The signal's half-life. If realised edge decays faster than your assumed horizon, the capacity calculation was wrong.
2. Fill rates against expectation. If passive fills come in materially below the simulated rate, the queue assumption was optimistic and the whole result rests on it.
3. Realised versus predicted costs. If costs exceed the model's, the edge may already be gone, and this is checkable within days.
4. Performance by regime. If the edge exists only in high volatility, or only at the open, say so, because the aggregate number is then misleading about most of the day.
5. Live against backtest divergence. Any systematic gap is information about which omitted cost is binding.

Key idea: Each of these is measurable in days rather than months, which is the strongest argument for deploying small and early rather than researching longer. The offline evaluation cannot resolve any of them, and a week of small live allocation resolves all five.

The course has been sceptical about directional prediction, and it is worth being clear that book models do earn their place, in applications where the cost hurdle does not apply.

• Execution. Deciding when to cross the spread and when to wait, informed by book state. The trade is happening anyway, so there is no cost hurdle to clear, only a cost to reduce.
• Market making. Quote placement and skew, where you earn the spread rather than paying it, which is the left end of the break-even curve.
• Short-term risk. Forecasting whether the next interval will be volatile or quiet, which is more predictable than direction and directly useful for sizing.
• Venue and routing choice. Where to send an order given current conditions across venues, again with no directional bet involved.
• Anomaly detection. Recognising unusual book states for risk and surveillance, where no trading decision follows at all.

Key idea: Every item on that list either avoids the cost hurdle or is on the earning side of it. That is the pattern: the book is far more useful for deciding how to trade, when you have already decided to, than for deciding whether the price will go up. The signal is real, small and short-lived, which is exactly the profile that suits execution decisions and does not suit directional bets.

1. The representation decides more than the architecture. Data level, snapshots against events, stationary features, sampling axis and depth are all fixed before a model is chosen, and each bounds what is achievable.
2. Architectures encode priors and add less than reported. Most published gaps over simple baselines are explained by weak baselines, unequal features and leaky evaluation rather than by depth.
3. The standard label is mostly nothing. At a one-event horizon, 92 percent of mid changes are zero, so accuracy without its majority-class baseline is uninterpretable.
4. Costs set a hurdle that accuracy hides. When the round trip costs as much as the move, break-even requires 100 percent accuracy.
5. Capacity is part of the result. A genuine edge that supports a few thousand shares may not cover its own infrastructure.

Key idea: The consistent thread is that the difficult parts of this problem sit outside the model. That is not a reason to avoid the work; it is a description of where the work is. A team that fixes the label, the validation and the cost model will get further with a gradient-boosted tree than a team that gets those wrong with anything.