Why Everything Gets Tested, and What a Test Actually Is

Start with the number that built the experimentation industry. Ron Kohavi, who ran experimentation platforms at Microsoft and Airbnb and co-wrote the standard text, Trustworthy Online Controlled Experiments, has reported repeatedly that at Microsoft only about a third of tested ideas improved the metrics they were designed to improve; roughly a third made no measurable difference, and a third actively hurt. Teams at other heavily-experimenting companies have published similar or harsher ratios, with well-designed, expert-reviewed ideas failing at rates that embarrass everyone's intuition.

Sit with what that means: the median confident product idea, backed by expertise, design review and conviction, does nothing or does damage. Nobody can tell which third an idea belongs to in advance, including the experts proposing it.

Key idea: experimentation is not a statistics hobby; it is the institutional admission that intuition about what helps users is wrong most of the time, made survivable by a machine that checks every idea cheaply. Companies like Booking.com and Airbnb run their product process on that admission, testing essentially everything user-facing.

The rest of this course is that machine: what it is, how it breaks, and the engineering that keeps its answers honest at scale.

The obvious objection: we have mountains of product data, why not just look? Ship the feature, watch the metrics.

Because observational product data is saturated with confounding, and product settings make it worse than most fields:

• Self-selection is everywhere. Users of the new feature differ from non-users before the feature does anything: they are the engaged ones who try things. Feature users retain better is the most reliably meaningless sentence in analytics.
• Everything moves at once. Marketing campaigns, seasonality, press coverage, a competitor's outage, three other teams' launches, all land in the same week as your release.
• The comparison you want does not exist in the data. The question is what would these same users have done without the change, and no dashboard contains that counterfactual.

Before-versus-after comparisons inherit every seasonal and news-cycle effect; users-who-adopted versus users-who-did-not inherits self-selection whole. Analysts patch with regression controls and matching, and the patches help, but they control only for what is measured, and engagement's causes are mostly unmeasured.

The experiment cuts the knot rather than untying it: create the counterfactual, by splitting genuinely comparable populations yourself.

The controlled experiment's design is almost insultingly simple: split users at random, show one group the change, hold the other on the current experience, compare the groups' metrics.

The power is entirely in the word random, and it is worth being precise about what it purchases. Random assignment makes the two groups statistically identical, in expectation, on every property, the measured ones and, crucially, the unmeasured ones: engagement, intent, device quality, mood, everything that confounds observational comparisons. Whatever influences the metric is split evenly between the arms, so a difference that emerges after the split has exactly one systematic explanation available: the treatment.

That is the entire trick. No causal model of user behaviour required, no list of confounders to enumerate, no statistical adjustment to argue about. Randomisation handles the confounders you did not think of, which, per the previous step, are most of them.

What remains is noise: any finite random split is slightly unbalanced by chance, so observed differences carry sampling variation, and the statistics of the field exist to answer one question, is this difference larger than chance-imbalance can plausibly explain? That machinery, and the peeking trap inside it, is lesson three. First, though, a decision that shapes everything downstream: random assignment of what, exactly?

Something gets randomly assigned to an arm, and the choice of that something is the platform's most consequential design decision.

Two forces push toward coarser units. Consistency: a user who sees the new checkout on Monday and the old one on Tuesday has an incoherent experience, and any metric accumulated across the flip-flop, retention, repeat purchase, is uninterpretable. Learning effects: anything a user notices or adapts to demands user-level stability.

The force pushing the other way is statistics: coarser units mean fewer independent observations. A million page views might be fifty thousand users or twelve cities, and the effective sample size for inference is the number of units, not the number of events, a fact that quietly invalidates analyses which randomise by user and then treat every page view as independent.

User-level randomisation, keyed on a stable identifier hashed with the experiment id, is the industry default, and the identifier's stability is itself an engineering problem: logged-out users, cleared cookies and cross-device identity all fray the unit, and every fray leaks users between arms.

An experiment needs a definition of better before it starts, and the experimentation literature is emphatic that choosing it after seeing the data is where integrity goes to die.

The structure that has become standard:

• One overall evaluation criterion, the OEC: the primary metric the experiment exists to move, ideally a proxy for long-term value rather than short-term activity. Kohavi's running example: sessions per user beats clicks per session, because clicks can rise while the product gets worse.
• Guardrail metrics: the things the change must not break while chasing the OEC: latency, error rates, unsubscribes, revenue, complaint volume. A win on the OEC with a broken guardrail is a loss with paperwork.
• Diagnostic metrics: the decomposition that explains movement, funnel steps, per-segment slices, consulted for understanding, not for verdicts.

Two disciplines make the structure real. The metrics, the analysis plan and the decision rule are written down before launch, because with dozens of metrics measured, some will move by chance, and post-hoc story-telling around whichever moved is indistinguishable from noise-worship. And the OEC is a standing institutional choice, not a per-experiment one: teams that pick a fresh success metric for each experiment have built a machine for confirming whatever they hoped.

Make the pieces concrete by configuring a real test. Hypothesis: showing delivery dates on product pages will increase completed orders.

1. OEC: orders per exposed user over the experiment window. Guardrails: page latency, support contact rate, refund rate. Diagnostics: add-to-cart rate, checkout entry, per-platform slices.
2. Randomisation unit: user, keyed on the stable account or device identifier, because the change is visible and buying decisions span sessions.
3. Population and trigger: users who view any product page after launch; exposure logs on first product page render, the earliest moment the treatment can act.
4. Split and duration: 50/50, sized by a power calculation, if orders per user needs 400,000 exposed users to detect the smallest lift worth shipping, and product pages see 60,000 eligible users a day, that is seven days minimum, rounded to two full weeks to cover weekly cycles.
5. Decision rule, written before launch: ship if the OEC lifts with significance at the horizon and no guardrail regresses; otherwise archive the result, including a null, in the experiment registry.

Nothing here is statistically deep, and that is the point: most of an experiment's trustworthiness is decided in this table, before a single user is assigned. The failure modes of each line, and the machinery that guards them, fill the rest of the course.

Calibration on the instrument's limits, before trusting it with the product:

And some questions the method structurally cannot answer: effects on non-users, brand perception in the market, anything requiring years to materialise, and changes that cannot be randomised, a rebrand, a price change everyone sees and discusses. The experiment is the sharpest instrument in the drawer, not the only one, and knowing the difference is part of the craft.

With the concept in place, the course turns to the machine itself, because between this lesson's clean theory and a trustworthy result at scale stands a pile of engineering: assignment that actually randomises, logging that actually captures exposure, and the detector that catches the machine lying, which is where we go next.