statistics

The most natural behaviour in experimentation, checking results daily and stopping when they look significant, quietly destroys the statistical guarantee everyone thinks they have. This lesson shows the peeking mechanism with honest arithmetic, then the fixes: fixed-horizon discipline, group sequential designs, and always-valid inference.

The companies famous for experimentation did not adopt it out of statistical enthusiasm: they adopted it because their own data showed most confident product ideas fail to improve the metrics they target. This lesson covers why observational product data misleads, what randomisation actually buys, the choice of randomisation unit, and the humbling base rates reported by the teams who measured.

The statistical core of backtest overfitting: why the best of many trials is inflated even when nothing works, how much to discount it, and why finance needs a far higher significance bar than the usual one.

A backtest is not a measurement of the past, it is a counterfactual about a world that never happened. Getting precise about that claim explains every way backtests mislead.

If the graph is an assumption, can you learn it from data instead? Conditional independence testing, the equivalence classes that limit what is knowable, and what changes when unmeasured confounders are allowed.

The central question of causal inference has a precise answer. The backdoor criterion, the front-door criterion, instrumental variables, and what to do when no identification strategy exists.

The formal machinery that makes causal questions answerable: structural causal models, graphs as assumptions you can inspect, the three ways variables become associated, and why prediction and intervention are different problems.

You randomized correctly and read the statistics honestly, and the experiment can still give the wrong answer, because the clean logic assumes things that are not always true. Learn how users affecting each other breaks the method, why a short test misjudges a long-run effect, how an average hides opposite effects in subgroups, and the reflex of distrusting results that look too good.

A perfectly randomized experiment can still hand you a confident, completely false result. Learn why, from the coin-flip nature of chance to the statistical significance that tells you less than you think. Covers p-values and what they really mean, why peeking at a running test wrecks it, how testing many things guarantees false wins, and why a result must be big enough to matter, not just real.

You change the button, sales rise, you conclude the button worked, and you may be completely wrong. Learn why comparing before and after is one of the most reliable ways to fool yourself, what a controlled experiment does differently, why randomization is the whole trick, and how the A/B test turns a guess about cause into something you can actually measure.

How to read a clinical trial result with discipline — the difference between absolute and relative risk reduction, what number-needed-to-treat captures, what confidence intervals actually mean, the hierarchy of evidence quality, and why statistical significance is not the same as clinical importance.

Build the vocabulary that underlies all of ML: sample spaces, discrete and continuous random variables, PMFs, PDFs, and CDFs. Then tour the key distributions — Bernoulli, Binomial, Categorical, Gaussian, Poisson, Exponential, Uniform — with their parameters, mean, variance, and exactly when each appears in practice.

Master the three numbers that summarize any distribution: mean, variance, and standard deviation. Derive linearity of expectation, understand covariance and correlation, then see why the Central Limit Theorem makes the Gaussian unavoidable — with a worked numeric example from scratch.

From raw data to defensible conclusions: derive Maximum Likelihood Estimators for Bernoulli and Gaussian, understand bias-variance in estimation, construct confidence intervals, and learn what p-values actually say — and don't say — including the most common misinterpretation that has corrupted thousands of papers.

Understand what it really means to update beliefs with data. Derive Bayes' theorem from first principles, dissect the roles of prior, likelihood, posterior, and evidence, work through a complete Beta-Binomial conjugate example numerically, and see why the base-rate fallacy trips up even experts.