Why before-and-after fools you

You want to know one thing: did my change cause the outcome? You made the sign-up button bigger, and sign-ups went up. Did the button do it?

The question feels trivial and is genuinely hard, because you are asking about causation, and causation cannot be seen directly. You only ever observe what did happen. To know if the button caused the lift, you would need to also know what would have happened without it, over the same period, with the same users, the same weather, the same everything. That parallel world, the counterfactual, does not exist. You cannot observe it.

Everything difficult about measuring cause comes from this single fact: the counterfactual is unobservable. You changed the button and the world moved on, and the version of events where you did nothing is gone forever.

So the entire discipline of experimentation is one question: since you cannot see the counterfactual, how do you construct a believable stand-in for it? Get that stand-in right and you can measure cause. Get it wrong, and you will confidently credit the button for a lift it had nothing to do with.

The tempting stand-in is the past: measure before the change, measure after, attribute the difference to the change. It is intuitive, it is everywhere, and it is one of the most reliable ways to fool yourself, because between before and after, everything else changed too.

Suppose sign-ups rose 20 percent after the new button. What else happened in that window? Maybe a marketing campaign launched, a competitor stumbled, it became the start of the month, a holiday passed, the product got mentioned somewhere, the weather turned. Any of these could have moved sign-ups. Before-and-after cannot separate your button from any of them, because they are all tangled together in time.

The formal name for the wreckers is confounding variables: things that changed alongside your change and could independently explain the outcome. Time is a confounding machine, because between two dates an unlimited number of things shift at once.

This is why "we changed X and the metric moved" is evidence of almost nothing. The metric moves constantly, for a hundred reasons. The question was never whether it moved. It was whether your change moved it, and a before-after comparison structurally cannot answer that.

The counterfactual is a parallel world you cannot visit. The controlled experiment's trick is to build one, in the present, by splitting your users into two groups at the same moment.

• The treatment group gets the change: the new button.
• The control group gets the old version: the old button.

Both groups live through the same period. The same campaigns, the same weather, the same day of the month, the same competitor news, everything external hits both groups identically, because they are running simultaneously. The only systematic difference between them is the thing you deliberately changed.

So when you compare the two groups' outcomes, every confounder that ruined before-and-after is now shared by both sides and cancels out. The marketing campaign lifted both groups; the holiday hit both groups. Whatever difference remains between treatment and control cannot be explained by those shared forces, because they affected both equally. It can only be explained by the one thing that differed: your change.

That is the whole conceptual move. Instead of comparing now to the past, where everything differs, you compare two groups in the same present, where only your change differs. The control group is your constructed counterfactual, living alongside the treatment.

One question decides whether the two groups are truly comparable: how do you split people into them? Get this wrong and the whole thing collapses.

Split them any way that correlates with the outcome and you have poisoned the comparison. Put desktop users in treatment and mobile in control, and any difference might just be desktop-versus-mobile, not your change. Let users choose, opt into the new version, and the eager, engaged people select themselves into treatment, so treatment wins because of who is in it, not what they got. This is selection bias, and it is fatal.

The answer is randomization: assign each user to treatment or control purely by chance, a coin flip. This is not a technicality; it is the entire foundation, and it works through a deep property.

Random assignment makes the two groups statistically identical on everything, not just the factors you thought to check, but every factor, including ones you have never heard of and could never measure. Age, device, mood, income, the hidden variable you do not know exists, all spread evenly across both groups by chance. So the groups start the same in every respect except the treatment, which means any difference at the end is caused by the treatment.

That is why randomization is magic: it controls for confounders you cannot even name, which no amount of careful analysis of observational data can ever do.

The A/B test did not come from technology. It is the randomized controlled trial, the method medicine uses to decide whether a drug works, applied to a product change. The parallel is exact and worth holding, because it imports a century of hard-won rigour.

A drug trial randomly splits patients: treatment gets the drug, control gets a placebo. Both groups then live through the same season, the same circulating illnesses, the same everything. The difference in recovery between them is the drug's effect, and randomization ensures the sicker or healthier patients did not cluster in one group.

Every element maps directly:

The reason medicine insists on this, rather than "we gave patients the drug and they got better", is precisely the confounding problem: people recover for many reasons, and only a randomized control isolates the drug's own effect. A product team running an A/B test is doing the same thing for the same reason, treating "does this change help" with the seriousness medicine treats "does this drug work".

Understanding that lineage is clarifying: an A/B test is not a growth-hacking gimmick. It is the gold standard for causal evidence, borrowed intact.

A trustworthy A/B test has a fixed shape, and every part guards against a specific way of being fooled.

1. A hypothesis, stated first. "The bigger button will raise sign-ups." Written before you look, so you cannot rationalise whatever you find afterward.
2. One change. Change the button only. Change the button and the headline and the colour together, and a win tells you nothing about which one did it. One variable per test.
3. Random assignment. Users bucketed by chance, so the groups are comparable on everything.
4. A primary metric, chosen in advance. The single number that decides it, named before the data arrives, so you are not free to hunt for any metric that happened to move.
5. A predetermined sample size and duration. How many users and how long, decided up front, for reasons the next lesson makes essential.
6. Run, then compare. Let it complete, then compare the primary metric between groups.

Each step blocks a particular self-deception. Deciding the metric and duration in advance stops you from stopping the moment the numbers flatter you. Changing one thing keeps the result interpretable. Stating the hypothesis first prevents inventing an explanation after the fact.

The pattern to notice: a rigorous experiment is mostly a set of commitments made before you see the data, precisely because a human looking at data will, without meaning to, find a story in it. The discipline is designed to protect you from your own pattern-matching.

Done right, an A/B test gives you something rare: credible causal evidence that your change moved a metric, for these users, in this context, right now. That is genuinely valuable and genuinely limited, and honest experimenters hold both.

What it does not give you:

• Why. It tells you the button worked, not the reason. The mechanism is your interpretation, and you can be wrong about it while the result stands.
• Whether it lasts. A change can win because it is new, then fade. A short test cannot see a long-run effect.
• Whether it generalises. It measured these users on this metric. Another audience, another context, another time may differ.
• The effect on things you did not measure. The button lifted sign-ups, and maybe it hurt something you were not watching. A test answers the question you asked, and is silent on the ones you did not.

The mature stance is that an experiment is a precise answer to a narrow question, not a verdict on your idea. It replaces "I think this helps" with "this raised this metric by this much, for these users, in this window, with this confidence", which is a far stronger claim and still a bounded one.

The next lessons are about the two ways this precision goes wrong: reading the numbers so as to fool yourself, and running tests in settings where the clean logic quietly breaks.

Before-after compares two time periods, so every external change confounds the result. The A/B test compares two randomized groups in the same period, so shared confounders cancel and only the change differs.