When your A/B test is lying anyway

The previous lessons gave a method: randomize, then read the statistics honestly. Done right, it delivers credible causal evidence. This lesson is the humbling part: even a perfectly executed experiment can hand you the wrong answer, because the clean logic rests on assumptions that are not always true.

The core assumption is quiet and load-bearing: that a user in the treatment group and a user in the control group are independent, that what happens to one does not affect the other. Randomization guarantees the groups are comparable at the start. It does not guarantee they stay isolated during the test.

When that independence holds, the whole framework works. When it breaks, and it breaks in some of the most important settings, an experiment can be run flawlessly and still mislead you, with all the statistics looking perfect.

This matters most precisely where the stakes are highest: social platforms, marketplaces, anything with the network effects from the earlier path. The very structure that makes those businesses valuable is the structure that breaks their experiments. So this final lesson is about knowing when to trust a clean result and when a clean result is quietly lying.

The most important way experiments break is interference: treating one user changes the experience of another, so treatment leaks into control and the two groups are no longer cleanly separated.

A concrete social example. You test a feature that makes people post more. Treatment users post more, as intended. But their posts appear in the feeds of their friends, some of whom are in control. Those control users now see more content, engage more, and maybe post more themselves. Control was supposed to be the untouched baseline, and the treatment has reached into it.

Now the comparison is broken in a specific, dangerous way. Treatment lifted control too, so the gap between them understates the true effect, and you conclude the feature did less than it really did. In other settings interference runs the other way and overstates the effect. Either way, the number is wrong.

This is not a rare edge case on a social platform; it is the default, because the entire point of such a product is that users influence each other. A marketplace has the same problem through competition: give treatment sellers a boost and they win sales that would have gone to control sellers, so control looks artificially worse and the boost looks better than it is. Wherever users interact, the independence assumption is not slightly wrong, it is structurally false.

You cannot statistics your way out of interference; you have to change what you randomize. The core fix is to randomize whole groups that interact internally rather than individuals.

If friends influence each other, split by clusters of friends: put entire social clusters into treatment or control together, so influence stays inside a single group and does not leak across the boundary. This is cluster randomization. The cluster, not the person, becomes the unit, so treatment effects that spread among connected people are captured within the treated clusters instead of contaminating control.

Other variants target other structures. Geographic tests put whole cities in treatment or control, so effects that spread locally, a marketplace, a delivery network, stay within a city. Time-based tests switch a whole system between versions in alternating windows and compare the periods.

Every one of these trades statistical power for validity. You have far fewer clusters than individuals, so it is harder to reach significance, but the answer you do get is not corrupted by interference. That trade is the right one when interference is present: a lower-powered honest answer beats a high-powered wrong one.

The meta-lesson is sharp: the hardest part of experimenting on a connected system is not the analysis, it is choosing the unit of randomization so that the thing you are measuring does not spill across your dividing line.

An experiment measures the effect during the test window, and for many changes the short-run and long-run effects differ, sometimes point in opposite directions.

Two mirror-image traps:

• The novelty effect. A new feature gets a burst of engagement simply because it is new. People click it to see what it is. A two-week test captures that curiosity spike and reports a win, but the effect fades as novelty wears off, and the long-run lift is smaller or gone. You shipped a result that was mostly "people poke at new things".
• The primacy effect. The reverse. A change to something familiar first disrupts habituated users, so it looks worse initially. Given time they adapt and the change proves good, but a short test, catching only the disruption, reports a loss and you kill an improvement.

The common cause is that behaviour has a time dimension the snapshot misses. The metric during weeks one and two is not the metric at month six, because users are still reacting to the change itself rather than settling into the new normal.

The defences follow directly: run tests long enough for novelty and disruption to settle; analyse new versus existing users separately, since novelty hits existing users and disruption hits the habituated; and for critical changes, keep a small long-term holdback group on the old version for months to measure the durable effect after the short-term reactions have washed out. The habit to build: always ask whether you measured a lasting effect or a temporary reaction to change.

An experiment reports an average effect, and an average can conceal that a change helped one group and hurt another, netting to a bland number that describes nobody.

Suppose a redesign is a small win overall. Underneath, it might be a big win for new users, who find the cleaner layout easier, and a real loss for power users, who lost a dense interface they relied on. The average, a slight positive, hides two strong and opposite effects. Ship on the average and you quietly harm your most valuable users while congratulating yourself.

The fix is to look within the experiment at how the effect varies across groups, new versus existing, mobile versus desktop, region, usage level. This is heterogeneous treatment effects, and it turns "did it work" into the sharper "who did it work for, and who did it hurt".

But this reopens a danger from the last lesson. Slicing by many segments is multiple comparisons, and slicing enough ways guarantees a spurious subgroup result. So there is a discipline: form your segment hypotheses before looking ("I expect this to affect new and existing users differently"), rather than dredging for any slice that shows something. A pre-planned segment split is analysis; a post-hoc hunt through fifty subgroups is fishing, and the same data supports both, so only your honesty about which you did separates a real finding from a fluke.

A rule from long experience with data, Twyman's law: any figure that looks interesting or surprising is usually wrong. Applied to experiments: a result far better than you expected is more likely a bug than a breakthrough.

The reasoning is calibration. Most changes have small effects; genuine large wins are rare. So when a test shows a massive, delightful lift, base rates say the most probable explanation is not that you found gold, but that something is broken in a way that manufactured the number.

The usual culprits are mundane and worth a checklist:

• Tracking errors. Conversions double-counted in one group, an event that fires only in treatment. The lift is in the logging, not the users.
• A broken control. The old version is failing, throwing errors, loading slowly, so treatment looks great only because control is silently degraded.
• Contaminated randomization. The split is not random, correlated with something, so the groups differed before the test began.
• A sample-ratio mismatch. You expected a 50/50 split and got 55/45, a red flag that something in the assignment is broken, and a strong hint the whole result is untrustworthy.

The healthy reflex is counter-intuitive and worth building: be most suspicious of your best results, and investigate a spectacular win as hard as a spectacular loss. A team that only scrutinises disappointing results and celebrates amazing ones will systematically ship bugs disguised as triumphs. Verify the wins you love before you believe them.

Three lessons converge on a mindset, not a formula, and it is worth stating plainly because it is the opposite of how experiments are usually sold.

Experimentation is presented as a machine that outputs truth: run the test, read the number, know the answer. The reality is that a controlled experiment is the best tool we have for measuring cause, and it is fragile, assumption-laden, and easy to fool yourself with. Both are true, and holding both is the whole skill.

The honest experimenter therefore does something specific: they treat a result as a claim to be attacked, not a verdict to be announced. Before believing a win they ask, could this be chance; did I peek; is it one of many comparisons; is it big enough to matter; could users be affecting each other; is it novelty; does it hide opposite effects in subgroups; is it too good to be true? Only a result that survives that gauntlet earns belief.

This is not cynicism about data; it is respect for it. The teams that get durable value from experimentation are not the ones that run the most tests. They are the ones most ruthless about not fooling themselves, because a confident wrong answer is worse than an honest "we do not know": the honest uncertainty keeps you looking, while the false certainty makes you act.

The entire path reduces to one habit. An experiment does not tell you what is true. It gives you evidence, and the quality of your conclusions depends entirely on how honestly you are willing to interrogate that evidence, especially when it tells you what you hoped to hear.

Randomization and honest statistics are necessary, not sufficient. Interference, time effects, hidden subgroups, and plain bugs can each corrupt a technically-perfect experiment.