Framing Execution as a Markov Decision Process

You hold a million shares to sell by the close. Selling them all at once walks the order book down and you receive a poor average price. Selling slowly reduces that impact and exposes you to the price drifting away while you wait. The trade-off between those two costs, and the classical schedules that resolve it, are the subject of the catalogue's Trade Execution course, and this course assumes them.

What that course does not cover is the framing this one needs.

Key idea: Execution is a control problem, not a prediction problem. You are not forecasting the price; you are choosing a sequence of actions whose effects persist, under uncertainty, with a terminal constraint that the whole position must be gone. Prediction is at most a component. The object being optimised is a policy.

The distinction that motivates this entire course is borrowed from control theory and is worth stating precisely.

Definition: An open-loop policy fixes the whole action sequence in advance, before observing anything. A closed-loop policy chooses each action as a function of the state observed at that moment.

A classical optimal schedule is open loop. It is computed once, from parameters estimated before trading starts, and it executes the same trajectory whether the book is thick or thin, whether volume arrived or did not, whether the spread widened or collapsed.

That is not a criticism of the mathematics, which is correct under its assumptions. It is a statement about what the mathematics was asked. Given a model where the state evolves independently of what you observe, the optimal policy genuinely is open loop, and the closed-form solution is exactly right.

Key idea: Reinforcement learning enters not because the classical answer is wrong but because it answers a question in which there was nothing to react to. The value of RL here is entirely the value of reacting, and that value can be measured.

Before building anything, it is worth knowing the most that adaptivity could ever be worth. Simulate a price path, then compare a fixed equal-slice schedule against an oracle that sees the entire future path in advance and schedules optimally against it, subject to the same quadratic impact cost.

The oracle cannot be beaten by any policy, adaptive or otherwise, because no real policy has more information than the future itself.

When impact is small the oracle saves two thirds of the cost. When impact is large it saves almost nothing.

Key idea: Impact cost is paid on the schedule, and a schedule that must sell a fixed quantity has little freedom once impact dominates. The headroom for any adaptive policy is not a constant; it is set by how much of your cost is impact and how much is timing.

Gotcha: This is why execution RL results reported on a few hundred episodes are uninformative regardless of the algorithm used, and why the field leans so heavily on simulators. It is also why apparent improvements so often fail to replicate: at these effect sizes, the difference between two policies over a short evaluation is mostly which paths they happened to see.

With the motivation established, the formulation is standard. A Markov decision process is a tuple of state, action, transition and reward.

Two entries deserve attention. The transition depends on your own action, which is what makes this control rather than prediction. And the terminal condition is a hard constraint rather than a preference, which shapes the whole policy: an agent that can be forced to dump at the end will learn to avoid ever being in that position.

Gotcha: Remaining inventory and time remaining must both be in the state. Omit either and the problem stops being Markov: the same book looks identical with 10 minutes left and with all day, and the correct action differs completely.

State design decides more than algorithm choice. The candidates fall into three groups with very different properties.

• Private state. Remaining inventory, elapsed time, your own fills so far. Always available, perfectly observed, and mandatory.
• Market state. Spread, depth at the touch, order book imbalance, recent traded volume, short-horizon realised volatility. These are the features that can genuinely inform a reaction, and book imbalance in particular carries short-horizon directional information.
• Signal state. Anything predictive you already have: a short-horizon alpha, a volume forecast, a news flag.

Gotcha: Including a directional signal quietly changes the problem. Pure execution takes the decision to trade as given and minimises cost; an agent rewarded for price improvement and handed a directional feature will learn to time the market, taking positions the mandate never authorised. If the agent can slow down when it expects the price to improve, it is running a small speculative book on top of your execution, and that exposure will not appear in any execution report.

The action space determines what the agent can express and how hard the learning problem is.

In practice: Deviation from a baseline is the design most likely to survive contact with a risk committee. The agent's output is a bounded adjustment to a schedule that is already understood and approved, so the worst-case behaviour is provable in advance, the comparison against the baseline is direct, and the agent can be clamped or disabled without leaving the desk without an execution algorithm.

The last row is where most academic work sits and where the simulator problem, the subject of the third lesson, becomes acute: an agent placing and cancelling limit orders needs a simulator that models whether those orders would have been filled, which historical data cannot answer.

Reinforcement learning was applied to this problem by Nevmyvaka, Feng and Kearns in 2006, in a paper presented at ICML titled "Reinforcement learning for optimized trade execution". It remains the standard reference point, and it is worth knowing what it did.

The setup was deliberately modest: a discrete state of remaining inventory and time, an action of where to place a limit order relative to the touch, and dynamic programming over historical order book data. The objective was implementation shortfall, the same benchmark the classical literature uses.

What it established was the framing rather than a production system: that execution decomposes into a sequence of state-dependent choices, and that a learned policy could improve on a fixed one within that data.

In practice: The two decades since have mostly changed the function approximator, from tabular dynamic programming to deep policy gradient methods, and the state representation, from a handful of buckets to raw book features. The hard part did not change. It is not the algorithm and it is not the network; it is whether the environment you trained against behaves like the market you deploy into.

Given the ceiling computed earlier, it is fair to ask what is left to win. The honest answer is that the reliable gains are not from predicting prices.

• Volume timing. Intraday volume has strong, stable seasonality. Trading in proportion to realised rather than forecast volume reduces impact, and this is a genuine prediction problem with far more signal than price direction has.
• Liquidity opportunism. Taking more when the book is unusually deep and the spread unusually tight is a reaction to an observable state, not a forecast.
• Queue and placement decisions. Where to sit in the queue, when to cross the spread rather than wait, when to cancel. These are high-frequency decisions where the state genuinely determines the right answer.
• Adapting to your own impact. Observing that your fills are moving the price more than expected and slowing down is closed-loop behaviour a static schedule cannot express.

Key idea: Every item on that list is a reaction to something observable, and none is a directional bet. That is the correct scope for execution RL. An execution agent that makes money by predicting prices has stopped doing execution, and the next lesson shows how easily a reward function invites exactly that.

Four preconditions, and failing any of them means the project will produce a number nobody should act on.

1. Impact must not dominate. If your order is a large fraction of daily volume, the chart above says the schedule is nearly forced and there is little for a policy to win. Adaptive execution pays off on medium orders, not on the largest ones.
2. You must have a credible simulator. Not historical replay: an environment that responds to your orders. This is the binding constraint in practice and gets its own lesson.
3. You must have enough independent episodes to measure the result. Thousands, not hundreds, and they must be independent, which consecutive days of the same name are not.
4. The mandate must be clear. Whether the agent is permitted to express a view on direction is a governance decision, and it has to be made before the reward function is written rather than discovered afterwards.

In practice: The most common failure is not a bad algorithm. It is a project that clears none of these four and reports a large improvement measured inside a simulator that could not have shown it anything else.