experimentation
4 free lessons tagged experimentation across Business. Each one is a short sequence of focused steps with narration and a five-question quiz at the end — take them in any order, no signup required.
CUPED and Interference: Faster Experiments, and When Arms Contaminate Each Other
Two advanced problems decide how much an experimentation platform is actually worth. Variance: most product metrics are so noisy that detecting small effects takes painful sample sizes, and CUPED buys the reduction with data you already have. And interference: in marketplaces and social products the arms affect each other, so the measured difference misstates what full launch will do.
Peeking: How Watching Your Experiment Ruins It
The most natural behaviour in experimentation, checking results daily and stopping when they look significant, quietly destroys the statistical guarantee everyone thinks they have. This lesson shows the peeking mechanism with honest arithmetic, then the fixes: fixed-horizon discipline, group sequential designs, and always-valid inference.
Assignment, Exposure, and the Smoke Detector Called SRM
Most wrong experiment results are not statistical subtleties; they are plumbing. This lesson covers how assignment actually works, hashing, not coin flips, why exposure must be logged at the moment of treatment, and the sample ratio mismatch check: the humble comparison of observed to expected group sizes that catches more broken experiments than any other single test.
Why Everything Gets Tested, and What a Test Actually Is
The companies famous for experimentation did not adopt it out of statistical enthusiasm: they adopted it because their own data showed most confident product ideas fail to improve the metrics they target. This lesson covers why observational product data misleads, what randomisation actually buys, the choice of randomisation unit, and the humbling base rates reported by the teams who measured.