The traps that make A/B tests lie

Here is the fact that everything in this lesson hangs on: even if your change does nothing at all, an A/B test will often show a difference between the groups, purely by chance.

Flip a fair coin a hundred times in two groups. The two groups will not tie. One will get more heads, by luck. Users are the same. Split any set of people into two random groups and measure anything, sign-ups, clicks, revenue, and the two groups will differ, even though nothing was done to either. Randomness does not produce equality; it produces approximate equality with a scatter around it.

So when a test shows the new button beat the old by 3 percent, there are always two explanations competing: your button genuinely helped, or the groups differ by chance and your button did nothing. Every result is haunted by this second possibility.

The whole of statistical testing exists to answer one question: is the difference I see bigger than what chance alone would routinely produce? Not "is there a difference", there is always a difference, but "is this difference too large to be luck". Everything that follows is machinery for answering that, and every trap is a way of getting the answer wrong.

The standard tool for this is the p-value, and it is one of the most misunderstood numbers in all of statistics. Get its meaning exactly right and half the traps disappear.

The p-value answers a very specific hypothetical: if my change truly did nothing, how likely is it that I would see a difference this big, or bigger, just by chance? A p-value of 0.03 means: if the button had no effect, a gap this large would happen about 3 percent of the time from luck alone. Small p-value: this result would be surprising if nothing were going on, so nothing going on becomes a less comfortable explanation.

Note carefully what it does not say. It is not the probability that your change works. It is not the probability that the result is a fluke. It is the probability of seeing data like this in a world where the change does nothing, which is a subtly but importantly different statement, and confusing the two is the single most common statistical error made about experiments.

The convention is to call a result statistically significant when the p-value falls below a threshold, usually 0.05. That number is a convention, not a law of nature, an agreed line for "surprising enough to act on", chosen by custom. There is nothing magic about 0.05, and treating it as a bright line between true and false is itself a mistake.

Because chance is always in play, any decision from an experiment can be wrong in two directions, and they trade off against each other.

• False positive (Type I error): you conclude the change works when it does not. You ship a useless button and believe it helped. The significance threshold controls this: a 0.05 cutoff means accepting roughly a 5 percent chance of crediting a change that did nothing.
• False negative (Type II error): the change genuinely works, but your test fails to detect it, so you discard a real improvement. This one is quieter and often ignored, and it is governed by statistical power: the probability your test catches a real effect of a given size.

The crucial link: these are in tension. Demanding very strong evidence to avoid false positives makes you miss more real effects, more false negatives. Being quick to declare winners catches more real effects and also more flukes.

Most teams obsess over false positives, the significance number, and completely neglect false negatives, power. That imbalance quietly kills good ideas: a real improvement tested with too few users produces an unconvincing result and gets thrown away, not because it did not work, but because the test was too weak to see it.

How many users do you need? This is not a detail to guess at afterward; it is a calculation to do before, and skipping it is why so many tests are inconclusive.

The smaller the effect you want to detect, the more users you need, and the relationship is steep. Detecting a huge 50 percent lift takes few users, because chance rarely fakes something that big. Detecting a realistic 1 percent lift takes a very large number, because a 1 percent gap is easily produced by luck, so you need enough data to distinguish a real 1 percent from a chance 1 percent.

This is why small sites struggle to test small changes: they simply do not have the traffic to tell a genuine tiny improvement from noise, and no cleverness fixes a sample that is too small.

Before running, you do a power calculation: given the smallest effect worth detecting and the level of certainty you want, it tells you how many users and how long you need. Run with fewer and you have built a test that cannot detect your effect even if it is real, guaranteeing an ambiguous result before you start.

The discipline this enforces is decisive: decide the sample size and duration up front, and do not stop early. The reason that matters so much is the next trap, and it is the one that ruins more real-world tests than any other.

Here is the trap that ruins more tests than any other, and it feels completely innocent. You start a test, and you watch it. Day one, treatment is up. Day two, they are even. Day three, treatment is up again and now significant. You stop and declare victory.

You have just corrupted the experiment, and the reason is precise. A running result wobbles: early on, with little data, chance swings it around a lot, and it crosses the significance line and back purely by luck. If you are allowed to stop the moment it crosses, you will stop on a lucky swing far more often than the 5 percent the threshold promises. Your real false-positive rate is no longer 5 percent; with continuous peeking it can climb toward a near-certainty of eventually seeing a fake win.

Think of it as flipping coins and being allowed to declare "this coin is biased" the instant heads is temporarily ahead. Wait for such a moment and you will always find one.

This is exactly why sample size and duration are fixed in advance. The commitment is not bureaucratic caution; it is what makes the significance number mean what it claims. Two honest fixes exist: run to your predetermined sample size and only then look, or use methods explicitly designed for continuous monitoring (sequential testing), which adjust the threshold to account for repeated looks. What you cannot do is use a fixed-sample p-value and peek, that combination silently inflates your error rate and hands you confident nonsense.

A close cousin of peeking, and just as common. Test twenty things at once, twenty button colours, twenty headlines, twenty metrics, and even if every one of them does nothing, you will very likely find at least one "significant" result, by chance alone.

The arithmetic is unforgiving. At a 5 percent threshold, each independent test has a 1-in-20 chance of a false positive. Run twenty such tests and, on average, one will come back significant even if all twenty changes are worthless. Find that one, announce it, and you have discovered pure noise and dressed it as a finding.

This is the multiple-comparisons problem, and it appears in disguises people do not notice:

• Testing one change against many metrics and celebrating whichever moved. Twenty metrics, same trap.
• Slicing a flat result by segment, mobile, country, new users, until some subgroup looks significant. Enough slices guarantee a winner.
• Running many experiments and only remembering the ones that "worked".

The defence is to be honest about how many chances you gave yourself. If you test many things, the bar for each must be higher (formal corrections tighten the threshold as the number of tests grows), or the apparent winner must be confirmed in a fresh test whose only job is to check that one thing. A finding fished out of many comparisons is a hypothesis, not a result, and the way you tell them apart is whether it survives a clean, pre-registered rerun.

The last trap is a pun the English language sets for you. Statistical significance and practical significance are different things, and a result can have one without the other.

Statistical significance means: the effect is probably real, unlikely to be chance. Practical significance means: the effect is big enough to matter. These come apart badly at large sample sizes. With millions of users, a test can detect a real effect so tiny it is worthless, a 0.01 percent lift that is genuinely there and completely irrelevant to your business. "Significant" in the report, meaningless in reality.

The corrective is to look past the p-value to two things it hides:

• The effect size. Not "is there an effect" but "how big is it". A real 0.01 percent is not worth shipping; a real 5 percent might be.
• The confidence interval. A range of plausible values for the true effect, which tells you far more than a single significant-or-not verdict. "Between 2 and 8 percent" and "between minus 1 and plus 9 percent" can both be technically significant and mean completely different things about whether to act.

The mature reading of an experiment is therefore never "was it significant". It is: how big is the effect, how uncertain is that estimate, and is the plausible range large enough to be worth acting on? A p-value alone answers none of those, and a decision made on the p-value alone is a decision made half-blind.

A difference between groups must clear several checks before it is trustworthy: not chance, not from peeking, not one of many comparisons, and big enough to matter.