backtesting

If your cost per round trip equals the move you are trying to capture, you need 100 percent directional accuracy to break even. This lesson computes that hurdle, replaces accuracy with metrics tied to a tradeable decision, and covers the capacity and latency limits that decide whether a real edge is worth anything.

Reinforcement learning needs an environment that responds to the agent, and historical market data is a fixed recording that does not. This lesson shows that replaying the same day gives a fill rate anywhere from 38 to 73 percent depending on an assumption the data cannot settle, and covers what to do about it.

Shuffled cross-validation reports 54 percent accuracy on data containing no signal, because neighbouring samples share their futures. This lesson builds purging, embargo and sample uniqueness weights, shows the illusion scaling with label overlap and vanishing when the fix is applied, and covers why a single walk-forward path is weak evidence.

Backtesting a language model signal has a defect no other signal has. The model was trained on text from the period being tested, so it may already know what happened next, and it uses that knowledge even when instructed not to. This lesson establishes the problem from the published evidence, shows why prompting does not fix it, and covers what does.

The edge that survives statistics still has to survive trading. Spread, market impact and the square-root law, why every strategy has a capital ceiling, and the research protocol that makes a backtest worth believing.

The statistical core of backtest overfitting: why the best of many trials is inflated even when nothing works, how much to discount it, and why finance needs a far higher significance bar than the usual one.

Look-ahead bias, survivorship bias, and point-in-time data. The errors that make a backtest wrong as a simulation, independent of any statistical question about whether the edge is real.

A backtest is not a measurement of the past, it is a counterfactual about a world that never happened. Getting precise about that claim explains every way backtests mislead.