The Simulator Problem: Why a Backtest Cannot Answer This

Supervised learning needs a dataset. Reinforcement learning needs an environment: something that accepts an action and returns a consequence. The difference is not a technicality.

A historical order book recording is a fixed record of what happened. It can tell you what the market did. It cannot tell you what the market would have done had your agent acted differently, and that counterfactual is precisely what every training step requires.

Key idea: This is the binding constraint in execution RL. Not the algorithm, not the network, not the compute. Everything else in the field is downstream of the question of what environment you trained against, and whether it could have shown your agent anything true.

The rest of this lesson is about how badly this bites, and what the available responses actually buy.

Take the most common setup: replay a recorded order book and let the agent trade against it. Three questions arise immediately and none has an answer in the data.

• Did my passive order fill? You joined a queue. Whether you reached the front depends on your position in it and on what happened ahead of you, and standard market data does not record either.
• What did my trade do to the price? Your marketable order consumed liquidity that, in the recording, was consumed by someone else or not at all. Every subsequent price in the replay is now wrong, and the error compounds across the episode.
• How would others have responded? Other participants react to order flow. Your orders were not in the recording, so nobody in it is reacting to you.

Gotcha: The third is the one people dismiss and it is the most damaging for a large order. The whole reason execution is difficult is that other participants detect and respond to persistent one-sided flow. A replay contains no such response, so it systematically understates the cost of exactly the behaviour you most need to avoid.

Key idea: That band is not measurement noise, it is a free parameter. Whoever wrote your simulator chose a point on this line, and the agent's entire passive-order strategy was learned against that choice. A policy tuned at 73 percent will behave very differently when the truth is 38.

Each tier answers a question the one above cannot, and each introduces a model whose errors the agent can exploit.

In practice: Match the tier to the action space. A participation-rate agent that only sends marketable orders needs an impact model and no queue model at all. An agent placing and cancelling limit orders needs the queue model, which is where the factor-of-two ambiguity lives, and that is a strong argument for restricting the action space until you have earned the right to widen it.

Add an impact model to a replay and something subtle happens: the agent's learned behaviour is now largely a consequence of that model's functional form rather than of anything in the data.

A linear impact model makes cost proportional to size, so splitting an order into two halves costs the same as one whole. A concave model, such as the widely used square-root form in which cost grows roughly with the square root of participation, makes splitting genuinely cheaper, and an agent trained against it will learn to split. The behaviour follows from the exponent, not from the market.

Gotcha: The agent will discover and exploit the exact shape you specified, including any region where the model is a poor approximation. If your impact function is fitted on orders up to 5 percent of volume and the agent finds an advantage at 30 percent, it has found an artefact of extrapolation, and it will look like a discovery.

The practical defence is to bound the agent's actions to the region where the model was estimated, and to treat any policy that concentrates its behaviour at those bounds as a warning rather than a result.

This is not specific to finance. It is the best-documented failure mode in applied reinforcement learning: given a simulator, an agent optimises the simulator, including the parts that do not correspond to anything real.

In execution the exploits have a recognisable shape.

• Trading into the model's blind spot. If cancellations are assumed always to be ahead of you, the agent learns to place passively and wait, because in the simulator waiting is nearly free.
• Exploiting the absence of reaction. With no participant responding to your flow, a policy that would be picked off in reality looks safe, so the agent learns to be more aggressive and more predictable than it should be.
• Riding the replay. If the recorded price path continues regardless of the agent, the agent can effectively see the future through the fixed path and learn to trade against it. This is look-ahead wearing a different costume.

Key idea: The diagnostic is a comparison, not an inspection. Train the same agent against two simulators that differ only in an assumption you cannot verify, such as the two ends of the queue band. If the policies differ materially, the policy is a property of the assumption rather than of the market, and no amount of further training will change that.

There is an alternative that avoids the simulator entirely. A trading desk generates enormous quantities of logged decisions and outcomes: orders sent, prices received, conditions at the time. Offline reinforcement learning learns a policy from that log without interacting with anything.

The difficulty is distribution shift. The log only contains actions your existing algorithms actually took, so a learned policy that recommends something different is being evaluated in a region where you have no data, and standard value estimates are wildly optimistic exactly there.

In practice: The offline family addresses this with pessimism: constrain the policy to stay near the logged behaviour, or penalise value estimates for actions poorly covered by the data. That deliberately limits how much better than your current algorithm the result can be, which is a real cost and an honest one. It also means the quality of your logging is the ceiling on the method, and logging that omits the state at decision time cannot support it at all.

Given that no offline evaluation is fully trustworthy, the honest answer is that the real measurement happens in production, carefully.

1. Shadow mode. Run the agent alongside the live algorithm, logging what it would have done without sending anything. Catches specification failures and crashes; tells you nothing about fills or impact, since nothing was sent.
2. Randomised allocation. Route a random fraction of orders to the agent and the rest to the incumbent. This is the only method that produces a genuine causal comparison, and randomisation is what makes it one.
3. Ramped exposure. Start at a small share of flow and increase it only as the measured difference clears the noise, which the first lesson established is a demanding bar.
4. Continuous monitoring against the incumbent, not against a backtest, since the backtest is the thing under suspicion.

Key idea: Randomised allocation is an A/B test, and it works for the same reason A/B tests work anywhere: it removes the confounding that every observational comparison suffers. It also generates exactly the exploratory data offline learning needs, so running it makes the next iteration better as well as measuring this one.

The experimental design, sizing and analysis are covered properly by the catalogue's A/B testing course, and the arithmetic there applies directly.

The right simulator is the cheapest one that can answer your question. Working from the question backwards avoids a great deal of wasted engineering.

In practice: The last row is the only one that is not a model. Everything above it is an argument about a model, and the discipline that matters most is knowing which row your claim is standing on.

A useful habit is to state, alongside any reported improvement, the assumption that would have to be wrong for the result to vanish. In execution RL that assumption is nearly always in the simulator, and naming it is usually enough to calibrate how much the number is worth.

Pulling the course together to this point.

1. The headroom is bounded and often small. Perfect foresight saves 66 percent when impact is negligible and almost nothing when impact dominates, so adaptive execution pays on medium orders rather than the largest ones.
2. The effect is buried in noise. Thousands of independent episodes are needed to measure even the maximum possible gain.
3. The reward is the specification, and the standard failures, under-execution, market timing, benchmark gaming, are all optimal behaviour under a reward that permitted them.
4. The simulator decides the answer. A single unverifiable assumption about queue position moves the fill rate by a factor of two, and the agent's policy is learned against whichever point you chose.
5. Only randomised live allocation settles it.

Key idea: None of this says the work is not worth doing. It says the difficulty is concentrated in the environment and the objective rather than in the learning algorithm, which is the opposite of where most effort goes. A team that spends its time on the simulator, the reward and the live experiment, using an unremarkable algorithm, will beat one that does the reverse.

The final lesson turns to market making, where the agent chooses both sides of the market and the inventory risk that follows is the whole problem.