Measuring Execution Honestly

Transaction cost analysis looks like an accounting exercise and is really a statistical one, which is why so much of it is unreliable.

The quantity being measured is a cost of perhaps 10 to 30 basis points. The noise it sits in is the instrument's price movement over the execution window, which for a liquid equity over a few hours is routinely 50 to 100 basis points.

So a single order tells you essentially nothing. The realised cost is dominated by whether the price happened to move while you traded, and that is close to a coin flip.

This has a consequence people resist: you cannot evaluate an execution from its outcome. A trade that finished at a great price in a falling market and a trade that finished badly in a rising one may have been executed identically well. Judging either from the number is judging the market's behaviour, not the desk's.

Only averages over many orders carry information, and the number required is larger than intuition suggests.

The sample size follows from the standard error of a mean, and the arithmetic is worth doing once because the answer is uncomfortable.

import math

def orders_needed(effect_bps, noise_bps, z=1.96):
    """Orders required to detect an effect of this size at ~95% confidence."""
    return math.ceil((z * noise_bps / effect_bps) ** 2)

orders_needed(effect_bps=5, noise_bps=60)    # 554
orders_needed(effect_bps=2, noise_bps=60)    # 3458
orders_needed(effect_bps=5, noise_bps=100)   # 1537

To establish that one algorithm is 5 basis points better than another, against 60 basis points of noise, takes over 500 orders per algorithm, executed under comparable conditions.

Most institutions do not have that. A desk trading a few hundred orders a year, across instruments with different liquidity and in different market regimes, cannot resolve a 5 basis point difference at all. Its quarterly TCA report is reporting noise.

This is the same problem the lesson Selection Bias and the Deflated Sharpe Ratio treats for strategies. Comparing several brokers or algorithms and picking the best one selects on a noisy measurement, and the winner's apparent advantage is mostly the maximum of a set of random draws.

Raw average cost is not comparable across orders, because the drivers of cost differ enormously between them. Comparing a desk's average this quarter to last quarter is mostly comparing the orders it happened to receive.

The standard fix is a pre-trade estimate: a model predicting what an order should cost, given its size relative to volume, the instrument's volatility, the spread and the intended horizon. Performance is then the difference between realised and predicted cost, not the realised cost itself.

That normalisation changes what is being asked. Instead of "was this cheap?" the question becomes "was this cheaper than an order like it should have been?", which is answerable and is the thing a desk actually controls.

It also creates an obvious incentive problem worth naming. If the same organisation builds the pre-trade model and is measured against it, a conservative model makes everyone look good. Serious programmes keep the estimate independent of the desk, and calibrate it on realised outcomes rather than on judgement.

A pre-trade model that is never wrong is not accurate; it is uncalibrated in the same way as a risk model with no breaches.

One measurement is unusually informative because it separates two costs that look identical in the total.

After a fill, watch the price for a defined interval. Two patterns are possible and they mean opposite things.

If the price reverts toward where it was, the move was temporary impact: liquidity was consumed and then replenished. That is a cost of demanding immediacy, and it responds to trading more slowly.

If the price continues in the direction of the trade, the move was permanent. Either the order carried information, or, more often, someone inferred the order's existence and traded ahead of the remainder.

def reversion(fills, mid_at_fill, mid_later):
    """Positive means the price came back: temporary impact.
    Negative means it kept going: leakage or genuine information."""
    signed = [(m0 - m1) if side == "buy" else (m1 - m0)
              for (side, _), m0, m1 in zip(fills, mid_at_fill, mid_later)]
    return sum(signed) / len(signed)

A desk whose fills show consistent post-fill continuation is leaking. That is actionable in a way a total cost number is not: reduce display size, randomise timing, spread across more venues.

The three components have three different owners, and reporting only the total makes it impossible to act on.

The delay component is usually organisational: how long between a decision and an order reaching a venue. It is often the easiest to reduce and the least examined, because no trading decision produced it.

The execution component is the trader's and the algorithm's, and it is the one everybody measures.

The opportunity component belongs to whoever chose the urgency, which is normally the portfolio manager rather than the desk. A high opportunity cost means orders are being worked too patiently for the signal behind them.

Splitting the number this way changes conversations. A desk defending a poor total when the loss was delay and opportunity cost is defending someone else's decision, and the fix lies outside its control.

Any measure used to evaluate people becomes a target, and execution benchmarks are unusually easy to hit without improving anything.

Measured against VWAP, a desk can concentrate trading in the part of the window where the price moved favourably. That is a directional bet dressed as execution quality, and it will average out to nothing while looking skilful in the periods it works.

Measured against arrival price, delay before the order arrives is free. Slow routing improves the score.

Measured on filled shares only, not completing is costless. The desk rests passively, fills the easy portion, and lets the rest expire.

Measured on spread capture, never crossing is optimal, regardless of what the order needed.

Each of these is a rational response to a specified objective, which is the point. The defence is not exhortation but measurement design: use implementation shortfall against the decision price, include unfilled quantity, and report the components separately so a gain in one is visible as a loss in another.

A benchmark that cannot be improved without genuinely improving execution is the only kind worth paying on.

Measurement only pays if it changes a decision, and that requires a closed loop rather than a report.

The working version has four stages. Estimate the cost before trading, from a model. Choose the schedule and algorithm using that estimate and the signal's urgency. Execute, recording the decision price, every fill, and the prices after each fill. Compare realised to estimated, attribute the difference to components, and feed the residuals back into the model.

The fourth stage is the one usually missing. Without it, the pre-trade model never improves, and the post-trade report is an archive rather than an instrument.

There is also a limit worth being honest about. Even a well-run loop resolves large differences slowly and small ones never. A programme that concludes broker A beats broker B by 2 basis points after 200 orders has concluded nothing, and acting on it is worse than not measuring, because it converts noise into a decision with switching costs attached.

The defensible use of TCA is to find gross problems, not to rank close competitors.

The last row is the one to insist on. A cost report without a sample size and a confidence interval is presenting a point estimate of a noisy quantity as though it were a measurement.

The honest summary of this cursus is that execution is a real discipline with a genuine optimisation at its centre, and that its results are much harder to verify than to produce. The schedule problem has a defensible answer. The question of whether your desk implements it well takes years of data to settle.

Which argues for spending effort on the parts that do not need statistics: reducing delay, avoiding benchmarks that reward the wrong behaviour, and not leaking the order in the first place.