CUPED and Interference: Faster Experiments, and When Arms Contaminate Each Other

The silent constraint on every experimentation programme is that product metrics are extraordinarily noisy. Revenue per user spans zero to enormous; sessions per week ranges from one to hundreds; and the effects worth detecting are small, a 1 percent lift on a core metric is often a large win at scale.

Detectable effect size shrinks only with the square root of the sample: to halve the minimum detectable effect, quadruple the users. For a metric whose standard deviation is large relative to its mean, powering an experiment to see 1 percent can demand millions of exposed users or months of runtime, and most companies have one of those at best.

This is not a pedantic concern; it decides what the platform can do. A team that needs six weeks per experiment runs eight experiments a quarter; a team that needs one week runs fifty. The entire compounding value of test everything rests on individual tests being cheap, which makes variance reduction, squeezing more sensitivity from the same users, among the highest-leverage machinery in the field.

Key idea: you cannot buy more users, but you can spend the noise smarter. The largest share of a metric's variance is usually not caused by the experiment at all: it is baseline differences between users that existed before the experiment started. Remove that share, and the same users answer harder questions.

The standard tool is CUPED, controlled-experiment using pre-experiment data, introduced by Deng, Xu, Kohavi and Walker in 2013 and now built into most serious platforms.

The observation: a user's metric during the experiment is strongly predicted by the same user's metric before the experiment. Heavy users were heavy last month; light users were light. That predictable component is pure noise for the causal question, the treatment cannot have caused pre-experiment behaviour, so subtract it:

where X is the user's pre-experiment value of the metric, and theta is chosen, as the covariance of Y and X over the variance of X, to minimise the adjusted variance. The variance shrinks by the squared correlation between before and during:

Because randomisation balanced the pre-period across arms, the adjustment cannot bias the comparison; it only removes noise. The practical numbers make the case: engagement-style metrics routinely correlate around 0.7 with their own pre-period, and 1 minus 0.49 halves the variance, the same sensitivity as doubling the user count, from a covariate that was sitting in the warehouse anyway. Deng and colleagues reported roughly 50 percent variance reduction on key metrics at Bing, which is why the technique spread everywhere.

The fine print: it needs a pre-period, so brand-new users contribute no reduction, mixed populations get partial benefit; and the covariate must be pre-treatment, always, using anything measured after assignment reintroduces the post-treatment conditioning bug from lesson two.

CUPED removes the predictable between-user noise; two humbler tools handle what it cannot, and both come with sharp edges.

Heavy tails. Revenue-style metrics are dominated by outliers: one whale in one arm can single-handedly manufacture or bury a significant difference. The standard treatment is winsorisation, capping each user's metric at a high percentile, the 99th or 99.5th, computed identically for both arms. The variance drops dramatically and with it the runtime; the price is honesty about what is now being measured: effects on capped revenue, not revenue. A treatment whose entire effect lives in whale behaviour is invisible after capping, so platforms report both views, capped for the verdict, uncapped for the sanity check, and investigate when they disagree.

Binary reformulations. Did the user order at all is far less variable than how much did they spend, and often nearly as decision-relevant. Choosing the binary or count version of a metric as the OEC, with the continuous version as a diagnostic, is free sensitivity whenever the decision truly hinges on participation rather than magnitude.

Gotcha: every one of these choices, the covariate, the cap percentile, the metric form, must be fixed before the data is seen, for exactly the reasons lesson three drilled: a cap chosen after noticing which arm the whale landed in is the peeking problem with a new costume.

Every technique in this course so far, the difference of means, the significance machinery, CUPED, rests on one quiet assumption with a formal name: SUTVA, the stable unit treatment value assumption. Each user's outcome depends only on their own treatment assignment, not on anyone else's.

For a button colour or a checkout flow, the assumption is fine: my experience does not reach into your session. But entire categories of products violate it structurally:

• Marketplaces: buyers compete for the same inventory; sellers compete for the same demand. Anything that makes treated buyers book more leaves fewer listings for control buyers.
• Social and communication products: treated users post more, and their control friends see the extra content; the treatment leaks along the social graph.
• Ride and delivery platforms: drivers are shared supply; treatment-induced demand moves pickup times for everyone.
• Shared infrastructure: a treatment that consumes more compute degrades latency for both arms equally, hiding its own cost.

When SUTVA breaks, the experiment still measures a difference, but the difference stops meaning what full launch will do, because control is no longer the world without the treatment: it is the world where the other half has the treatment, which is a different counterfactual entirely.

The canonical interference failure deserves its own walkthrough, because it inflates results in the direction teams most want to believe.

The same illusion appears wherever arms share a finite resource: ad auctions, where treated campaigns outbid control campaigns; notification experiments competing for the same attention; matching systems dividing the same candidate pool. The tell is a fixed pie anywhere in the loop, and the first diagnostic question for any marketplace experiment is: could this arm's gain be the other arm's loss?

Interference cannot be patched in analysis; it has to be contained in the design, by drawing the randomisation boundary around the interference instead of through it.

Cluster randomisation assigns whole markets, cities, or social communities to one arm: interference flows freely inside each cluster, where everyone shares an arm, and the comparison runs between clusters. The price is statistical: the effective sample size is the number of clusters, not users, and clusters are few, heterogeneous, and expensive to balance.

Switchback designs, standard in ride and delivery platforms, randomise time instead of users: the whole market flips between treatment and control in blocks, hours or days, and the comparison is across periods. Interference within each period is contained because everyone shares an arm; the cost is temporal correlation, carryover effects between adjacent blocks, and sensitivity to time-of-day confounding, managed with washout gaps and careful block lengths.

Graph-aware designs cut social networks into low-connectivity components before assigning arms, minimising cross-arm edges. And two-sided designs randomise both sides of a marketplace in a crossed pattern, which lets the interference itself be estimated rather than merely feared.

The general recipe: identify the channel the treatment leaks through, inventory, graph edges, shared supply, and choose units the channel does not cross.

The course's machinery, compressed into the questions a competent experiment answers before launch:

Eight questions, one page, and the difference between an experimentation programme and a random number ceremony.

The arc of the course closes where it began: with the base rate. Most ideas fail, which is precisely why the machine that finds the exceptions must be trustworthy, fast, and honest about its own limits. The companies that test everything did not build that machine because they love statistics; they built it because they measured their own judgement and took the result seriously. That, in the end, is the transferable habit.