Market Making: Inventory, Adverse Selection, and What RL Adds

Execution has a target: sell a million shares by the close. Market making has no target at all. You quote a bid and an ask continuously, hoping to buy at one and sell at the other, and your desired position is zero at all times.

That inverts the control problem. In execution, inventory is the thing you are deliberately working down. In market making, inventory is an accident: every fill leaves you holding something you did not want, in a direction you did not choose.

Key idea: A market maker is paid the spread for providing immediacy and is exposed to two risks in return. Inventory risk is the price moving against an accidental position. Adverse selection is the fact that the people most eager to trade with you are disproportionately the ones who know something you do not. Almost everything in market making is managing one of those two.

Quote a fixed spread symmetrically around the mid and your inventory performs a random walk. Buys and sells arrive at roughly equal rates, so the position drifts with no restoring force, and a random walk's expected distance from zero grows without bound.

Worse, the drift is correlated with the thing that hurts. If the price is falling, sellers hit your bid more often than buyers lift your offer, so you accumulate a long position precisely while the asset is declining. The inventory you acquire is systematically the inventory you least want.

Gotcha: This is not a tail risk that shows up occasionally. It is the ordinary behaviour of symmetric quoting, and it means the spread you earn per round trip is being paid against a position whose size is growing over the session. A market maker who only manages the spread and not the inventory is running an unhedged directional book that they did not choose the direction of.

The classical answer, from Avellaneda and Stoikov's 2008 paper "High-frequency trading in a limit order book" in Quantitative Finance, is to stop quoting around the mid and start quoting around a reservation price that moves with your inventory.

$r = s - q\,\gamma\,\sigma^2 (T - t)$

where $s$ is the mid, $q$ your inventory, $\gamma$ risk aversion, $\sigma$ volatility and $T - t$ the time remaining. Long inventory pushes the reservation price down, so both quotes shift down: your offer becomes more attractive and your bid less so, and the flow you attract is skewed toward reducing the position.

Definition: The reservation price is the mid-price at which you would be indifferent to holding your current inventory. It is not a forecast. It is your own indifference point, and quoting around it makes your quotes express how much you want to be flat.

The two adjustments are separable: the skew controls the direction of the flow you attract, and the spread width controls how much you earn per fill and how many fills you get.

Simulating 20,000 sessions of 200 quoting intervals each, with fill probability falling exponentially in distance from the mid, gives the trade-off directly. Both series are indexed to the symmetric case.

At the strongest skew, average inventory exposure falls by 63 percent and the maximum position across sessions drops from 42 units to 16, for a profit cost of 2.5 percent.

Key idea: Inventory control is cheap. The two lines diverge immediately, which means the first units of skew are nearly free: they cut exposure substantially and cost almost nothing, because the flow they redirect was arriving anyway. A market maker who is not skewing is leaving the easiest risk reduction in the business unclaimed.

The distinction to hold is between two kinds of counterparty. Uninformed flow trades for reasons unrelated to short-term price, a fund rebalancing, a retail order, an index change, and paying the spread to that flow is the business. Informed flow trades because it expects a move, and every fill against it is a small loss disguised as a small gain.

Key idea: The spread is not a fee, it is compensation for a risk. It must be wide enough that the profit from uninformed flow exceeds the losses to informed flow, which means the correct spread depends on the mixture of counterparties rather than on your costs. When the mixture shifts toward informed flow, the correct response is to widen, and a market maker who does not is being picked off at their own quoted price.

Detecting the shift is where the observable state earns its place, and the useful indicators are all short-horizon and market-structural.

In practice: The last row is why market makers widen or withdraw before scheduled announcements. It requires no prediction at all: everyone knows information is coming, so anyone trading through it is more likely than usual to be trading on it.

The other rows are exactly the kind of state a learned policy can react to, and this is the strongest case for RL in market making. The mapping from these signals to the right quote adjustment is nonlinear, interacts across signals, and is not something a closed-form model derived under a fixed arrival rate can express.

The closed-form solution is genuinely useful and it is derived under assumptions worth naming, because the gap between them and reality is exactly the space a learned policy occupies.

Key idea: Every row is a place where reacting to observable state could improve on a fixed rule, and that is the honest scope for RL here. It is a larger scope than in execution, because a market maker faces a continuing stream of decisions rather than one order with a bounded ceiling on adaptivity.

The last row is the least discussed and matters most at scale. Managing inventory name by name ignores that a long in one name may hedge a short in a correlated one, and the portfolio-level version of this problem is genuinely hard.

The MDP is a continuing one rather than episodic, and every component shifts relative to execution.

Gotcha: The reward has a term for the mark-to-market on inventory, and that term is a directional profit and loss. An agent maximising it will learn to take positions on purpose, holding inventory when it expects the price to rise. That may be acceptable, and it must be a decision rather than an accident, because it converts a market-making agent into a market-making-and-speculation agent whose risk sits outside the mandate.

The absence of a terminal condition is the other structural difference. Execution's forced liquidation anchors the value function; market making has no such anchor, so the risk penalty on inventory is doing all the work of keeping the agent from drifting into an arbitrarily large position.

Everything in the previous lesson applies here and is harder, because a market maker lives entirely on the passive side of the book.

An execution agent can be restricted to marketable orders, in which case fills are certain and only impact needs modelling. A market maker cannot: quoting is the strategy, so every single decision depends on the fill question whose answer ranges over a factor of two.

It is also more exposed to the absent-counterparty problem. Adverse selection is by definition other participants choosing to trade against you because of what they know. A replay contains no participants who can make that choice, so a naive simulator shows a market maker earning the spread cleanly and almost never being picked off.

Gotcha: The result is that market-making backtests are systematically optimistic in a specific direction: they capture the revenue, which is mechanical, and omit the main cost, which requires an adversary. A strategy showing a smooth equity curve in replay is showing you a world with no informed traders in it.

This pushes market-making RL toward agent-based simulation, where informed participants can be modelled explicitly, and toward the same conclusion as before: the live randomised experiment is what settles it.

Four lessons, and the argument compresses into five statements.

1. The classical solutions are open-loop, and RL's entire value is closing the loop. That value is real and it is bounded, and in execution the bound collapses as impact grows.
2. The effect sizes are small relative to the noise. Thousands of independent episodes are needed to measure them, which rules out most published evaluations.
3. The reward is the specification. Under-execution, market timing and benchmark gaming are all optimal behaviour under rewards that permitted them, and all are detectable on paper before training.
4. The simulator determines the answer. One unverifiable queue assumption moves the fill rate by a factor of two, and a replay with no reacting participants systematically flatters passive strategies.
5. Only a randomised live experiment settles it.

Key idea: The difficulty in this field is concentrated in the environment and the objective, not in the learning algorithm. That is unusual, and it is why progress here looks less like better networks and more like better simulators, tighter mandates and honest experiments. The teams that get results are the ones that treat the algorithm as the easy part.