Reward Design: Where Execution Agents Go Wrong

Everything an agent does is a consequence of the reward function. Not the intent behind it, not the docstring above it, not the slide describing the project. The reward is the only channel through which your objective reaches the policy, and anything absent from it is something the agent is free to sacrifice.

Execution is unusually exposed to this. The objective has several components that pull against each other, cost, risk, completion, and market conditions, and only some of them are naturally expressible as a per-step number.

Key idea: Every failure in this lesson is an agent behaving optimally. There is no bug, no instability, no insufficient training. The policy is a correct solution to the problem that was actually posed, and the gap between that problem and the intended one is the entire subject.

The standard objective is implementation shortfall: the difference between what you realised and what the position was worth when the decision was made.

$\text{IS} = \sum_i q_i \, p_i \;-\; X \, p_{\text{arrival}}$

for a sell of $X$ shares filled in slices $q_i$ at prices $p_i$. It is the right economic objective, and as a reinforcement learning signal it has an awkward property: it is naturally terminal. You do not know the shortfall until the order is finished.

A terminal-only reward creates the credit assignment problem in its hardest form. An episode might contain 400 decisions and one scalar at the end, and the learner must apportion blame across all of them from a signal whose variance, as the previous lesson computed, is many times larger than the effect being optimised.

In practice: This is why almost every implementation reaches for a per-step reward, and why almost every implementation then introduces the failures in the rest of this lesson. The pressure to densify the reward is real, and it is where the specification quietly changes.

The threshold is computable. With quadratic impact $\eta q^2 / T$ over horizon $T$ and a per-share penalty $\lambda$ on unexecuted inventory, the agent's optimum is

$q^* = \min\left(X,\ \frac{\lambda T}{2\eta}\right)$

so full execution requires $\lambda \geq 2\eta X / T$: the penalty must exceed the marginal impact cost of the final share.

Gotcha: The threshold depends on order size, horizon and current liquidity. A penalty tuned on medium orders in calm markets will be below the threshold for large orders in thin ones, so the agent will silently under-execute in exactly the conditions where completion matters most.

The relationship is linear right up to the threshold and then flat. At half the required penalty the agent executes a quarter of the order. There is no regime in which a slightly-too-small penalty produces slightly-too-little execution: the shortfall is proportional and can be enormous.

In practice: Make the terminal condition a constraint rather than a cost. Force liquidation of any residual at the horizon at a genuinely punitive price, so that leaving inventory is never on the efficient frontier at all. A constraint the agent cannot trade against is more robust than a penalty it can, and it removes the need to tune a number whose correct value moves with market conditions.

Reward the agent purely on price achieved and give it any feature carrying short-horizon directional information, and it will learn to time the market. It will hold back when it expects a better price and accelerate when it does not.

That looks like an improvement in every execution report, because execution reports measure price against a benchmark. What is actually happening is that the agent has taken a directional position: by deviating from the schedule it is long or short relative to the mandate, and the resulting profit and loss is a bet, not a saving.

Gotcha: These are indistinguishable in the metric almost everyone uses. A shortfall improvement tells you the agent got a better price and says nothing about whether it did so by trading better or by taking a position. Separating them requires attributing the gain to schedule deviation and marking that deviation as a position, which very few execution reports do.

Benchmarks are chosen for measurability, and every measurable benchmark can be influenced by the thing being measured.

• Volume-weighted average price. An agent measured against VWAP over its own trading window can improve its score by trading when its own volume dominates the benchmark, pulling VWAP toward its own fills. The score improves and nothing real does.
• Arrival price. The benchmark is fixed at the start, which is better, but it makes the score a function of what the market did afterwards. An agent evaluated on it is partly being scored on luck, which adds variance rather than bias, and variance is what the previous lesson showed you cannot afford.
• Interval VWAP with a fixed window. Trading right at the window edges can shift which prints fall inside the benchmark.

Key idea: This is Goodhart's law with a specific mechanism. In most settings a measure stops being a good measure because people optimise it. Here the agent is also a participant in producing the measure, so it can move the target rather than merely gaming its own score. Any benchmark computed from a window the agent trades in is influenceable by construction.

Given the pressure to densify a terminal reward, the question is which dense rewards are safe. There is an exact answer, from Ng, Harada and Russell at ICML 1999, "Policy invariance under reward transformations".

A shaping term is safe if and only if it is potential-based: derived from a function $\Phi$ of the state alone, added as the difference between the potential at the next state and the current one.

$F(s, a, s') = \gamma\,\Phi(s') - \Phi(s)$

Adding such a term leaves the optimal policy unchanged for any $\Phi$, in any environment. The intuition is that the added rewards telescope: around any loop the contributions cancel, so no cycle can be made artificially profitable.

Key idea: Any dense reward not of this form can change the optimal policy, and the paper's point is that the well-known bugs in shaping are exactly the non-potential-based cases. So the discipline is simple: shape by defining a potential over states, never by handing out bonuses for actions you approve of.

A reasonable potential for execution is a function of remaining inventory against the schedule, which rewards progress without rewarding any particular way of making it.

Minimising expected cost alone produces an agent indifferent to variance, which no desk wants. The standard remedy is a mean-variance objective, penalising the risk carried while inventory remains.

$R = -\text{cost} \;-\; \kappa \sum_t \sigma_t^2 \, I_t^2$

where $I_t$ is inventory at time $t$ and $\kappa$ is risk aversion. The penalty is quadratic in inventory and scales with prevailing volatility, so the agent trades faster when it is holding more or when the market is more dangerous.

In practice: $\kappa$ is the same risk-aversion parameter the classical schedules expose, and the sanity check is worth running: with $\kappa$ set to match, and adaptivity switched off, the learned policy should reproduce the classical trajectory. If it does not, the disagreement is a bug in the environment or the reward, not an insight, and you have found it cheaply.

That comparison is the most useful test in this whole area, because it is the one case where you know what the right answer looks like.

Some requirements are not preferences to be traded off. Do not exceed 20 percent of volume. Do not cross the spread more than $n$ times a minute. Never leave inventory overnight. Encoding these as penalties invites the agent to price them and pay them when the reward makes it worthwhile.

Gotcha: A penalty is a price. The moment a rule is expressed as a number subtracted from the reward, you have told the agent what you are willing to sell that rule for, and a sufficiently capable agent will find the situations where buying it is worthwhile. For anything that is genuinely a rule, mask the action or wrap the policy, and keep it out of the reward entirely.

Before training anything, put the proposed reward through these.

1. What is the cheapest way to score well? Answer it adversarially, as the agent would. If "do nothing" or "trade at the very end" scores acceptably, the specification is wrong.
2. Is completion a constraint or a cost? If it is a cost, compute the threshold at which under-execution becomes optimal and check your penalty clears it in the worst conditions, not the average ones.
3. Can the agent influence its own benchmark? If the benchmark is computed over a window the agent trades in, it can.
4. Is every shaping term potential-based? If not, it can change the optimal policy, and you should assume it has.
5. Are genuine rules expressed as constraints rather than penalties? Anything you would not sell at any price must not have a price.
6. Does the reward permit directional positions? If yes, that is a mandate decision that needs signing off, not a modelling detail.
7. Does it reproduce the classical solution in the classical special case? If not, stop and find out why.

Key idea: Every one of these is answerable on paper, before a single episode is run. Reward specification is the cheapest stage of this work to get right and the most expensive to get wrong, because a mis-specified reward produces a confident, well-trained, thoroughly-evaluated agent that is solving a different problem.