ab-testing
7 free lessons tagged ab-testing across Business. Each one is a short sequence of focused steps with narration and a five-question quiz at the end — take them in any order, no signup required.
CUPED and Interference: Faster Experiments, and When Arms Contaminate Each Other
Two advanced problems decide how much an experimentation platform is actually worth. Variance: most product metrics are so noisy that detecting small effects takes painful sample sizes, and CUPED buys the reduction with data you already have. And interference: in marketplaces and social products the arms affect each other, so the measured difference misstates what full launch will do.
Peeking: How Watching Your Experiment Ruins It
The most natural behaviour in experimentation, checking results daily and stopping when they look significant, quietly destroys the statistical guarantee everyone thinks they have. This lesson shows the peeking mechanism with honest arithmetic, then the fixes: fixed-horizon discipline, group sequential designs, and always-valid inference.
Assignment, Exposure, and the Smoke Detector Called SRM
Most wrong experiment results are not statistical subtleties; they are plumbing. This lesson covers how assignment actually works, hashing, not coin flips, why exposure must be logged at the moment of treatment, and the sample ratio mismatch check: the humble comparison of observed to expected group sizes that catches more broken experiments than any other single test.
Why Everything Gets Tested, and What a Test Actually Is
The companies famous for experimentation did not adopt it out of statistical enthusiasm: they adopted it because their own data showed most confident product ideas fail to improve the metrics they target. This lesson covers why observational product data misleads, what randomisation actually buys, the choice of randomisation unit, and the humbling base rates reported by the teams who measured.
When your A/B test is lying anyway
You randomized correctly and read the statistics honestly, and the experiment can still give the wrong answer, because the clean logic assumes things that are not always true. Learn how users affecting each other breaks the method, why a short test misjudges a long-run effect, how an average hides opposite effects in subgroups, and the reflex of distrusting results that look too good.
The traps that make A/B tests lie
A perfectly randomized experiment can still hand you a confident, completely false result. Learn why, from the coin-flip nature of chance to the statistical significance that tells you less than you think. Covers p-values and what they really mean, why peeking at a running test wrecks it, how testing many things guarantees false wins, and why a result must be big enough to matter, not just real.
Why before-and-after fools you
You change the button, sales rise, you conclude the button worked, and you may be completely wrong. Learn why comparing before and after is one of the most reliable ways to fool yourself, what a controlled experiment does differently, why randomization is the whole trick, and how the A/B test turns a guess about cause into something you can actually measure.