Benchmarks That Hold Up, and Knowing When to Stop

The purpose of a benchmark is to support one claim: this version is faster than that version. That is a causal claim about two variants, which makes it an experiment, and experiments have requirements.

The controls a benchmark needs:

• One variable. Change the code under test and nothing else. Not the machine, not the data, not the dependency versions, not the time of day.
• Enough repetitions. A single run is a sample of one drawn from a noisy distribution, and single-run comparisons are how teams convince themselves of improvements that do not exist.
• Reported variance. "120 ms" is not a result. "120 ms, with a standard deviation of 3 ms across 50 runs" is, because it tells the reader whether a 5 ms difference means anything.
• A realistic workload. Measuring the case you optimised for, rather than the case users produce, guarantees a favourable answer and no benefit.

Key idea: the discipline the catalogue's A/B testing course applies to product experiments applies identically here, and for the same reason. A comparison with an uncontrolled variable or a sample of one is not evidence, whatever number it printed.

In runtimes with just-in-time compilation, the same code gets faster as it runs, which makes the first measurements systematically wrong.

What changes during the early executions: bytecode is interpreted before it is compiled to machine code, the compiler needs execution counts before it optimises a method, inlining decisions depend on observed call patterns, caches are cold, and connections are not yet established.

The consequence is that a naive benchmark measuring the first hundred iterations reports the interpreter's performance, not the program's, and a comparison between two variants may be comparing two different compilation states rather than two implementations.

In practice: run a warmup phase whose results are discarded, then measure. Purpose-built benchmarking harnesses do this for you and handle several other traps besides, which is a strong argument for using one rather than a hand-rolled timing loop.

Gotcha: ahead-of-time compiled languages are not exempt. Caches, memory allocator state, branch predictor training and file system caches all warm up too. The effect is smaller than a JIT's but large enough to swamp the small differences microbenchmarks typically try to detect.

The most embarrassing benchmarking failure is measuring code that did not run, and it happens more often than anyone admits.

Related traps in the same family: constant folding, where inputs known at compile time let the optimiser precompute the answer; and loop hoisting, where a computation that does not vary is moved out of the loop and performed once.

Suppose the benchmark is written perfectly: warmed up, repeated, variance reported, results consumed. It can still produce a conclusion that does not survive contact with the real system.

The reasons are all about isolation:

• Perfect cache conditions. A tight loop over one small data structure keeps everything in L1 cache. In the real application that structure competes with everything else, and the cache hit rate is nothing like the benchmark's.
• Unrealistic branch prediction. Repeating the same input teaches the predictor the pattern perfectly; production input is varied and mispredicts.
• No contention. The benchmark runs alone. Production runs alongside other threads, other requests and other processes competing for the same caches and memory bandwidth.
• Amdahl again. A function made three times faster in isolation may be two percent of the request, and the system-level difference is unmeasurable.

Key idea: microbenchmarks answer "which of these implementations is faster in isolation", which is a genuine question when choosing between two algorithms. They do not answer "will this make my application faster", and the only instrument that answers that is a measurement of the application.

Both are useful. Confusing the first for the second is how weeks get spent making a component fast that nobody was waiting on.

Performance decays gradually. No single change makes an endpoint slow; forty changes each adding two percent do, and nobody notices because no individual pull request looked wrong.

The defence is automated regression detection, and its central difficulty is noise. Shared continuous integration machines have noisy neighbours, thermal throttling and variable virtualisation, so run-to-run variance can easily exceed the regressions you want to catch. A naive threshold either fires constantly and gets ignored, or is set so wide that real regressions pass.

The approaches that work in practice:

• Compare within a run, not across days. Measure the old and new code in the same job on the same machine, so machine-level noise affects both.
• Alert on sustained trend, not single points. A rolling comparison over several commits distinguishes a real step change from one unlucky run.
• Use dedicated hardware for the benchmarks that matter, accepting that this is a real cost for a real signal.
• Prefer counting to timing where possible. Query counts, allocation counts and instruction counts are far less noisy than wall time and catch a large share of regressions.

In practice: that last point is underused. Asserting that an endpoint issues at most N queries is a stable, fast test that catches the N+1 regression permanently, and it never flakes because of a busy build machine.

Optimisation has diminishing returns and rising costs, and the discipline is recognising the crossing point rather than continuing until interrupted.

The signals that you have arrived:

• The target from lesson one is met. This is the designed stopping condition, and honouring it is the whole reason for writing it down.
• The remaining profile is flat. No single item is more than a few percent, so Amdahl's law says nothing left is worth attacking individually.
• The next win requires an architectural change. A rewrite or a redesign is a different decision with a different budget, not a continuation of this one.
• The code is getting harder to read for single-digit percentages. Every optimisation past this point is a permanent maintenance cost paid for an unmeasurable user benefit.

Gotcha: performance work is unusually satisfying, which is exactly why it overruns. There is a clear number that goes down, immediate feedback, and puzzle-like problems. Those properties make it easy to keep going long after the user-visible benefit has stopped, while the codebase accumulates cleverness that the next engineer will be afraid to touch.

The honest test at that point is whether a user could tell the difference. If nobody can perceive the improvement and no cost has fallen, the work has become a hobby with a budget.

Sustained performance comes from ordinary practices rather than heroic optimisation projects, and the four lessons of this course reduce to a handful of them.

• Measure before changing, always. Intuition about where time goes is wrong often enough that acting on it is gambling.
• Apply the arithmetic first. A component's share of total runtime caps what optimising it can achieve, and that ceiling decides where effort goes.
• Match the tool to the symptom. Idle machine and slow requests means waiting, so profile wall-clock time or trace; busy machine means computing, so profile CPU and read the widest boxes.
• Know the six causes. Waiting, query multiplication, allocation, contention, memory layout and serialisation account for the overwhelming majority of real slowness.
• Benchmark like an experiment, and treat any result that flatters you as suspect until it survives a check.
• Guard what you fixed. A regression test on query counts or a tracked p95 keeps the improvement, and without one it will erode two percent at a time.

Key idea: the goal was never a fast function. It is a system whose performance is understood, measured, and defended, so that when it does get slow, finding out why is an afternoon rather than an archaeology project. That understanding is the durable asset; any individual optimisation is just one of its consequences.