Assignment, Exposure, and the Smoke Detector Called SRM

The naive image of random assignment, flip a coin when the user arrives, store the result, fails several requirements at once: it needs a write on every first exposure, a lookup on every subsequent one, and it cannot be reproduced after a data loss or across systems.

Production assignment is deterministic hashing. Concatenate a stable user identifier with the experiment's identifier and a salt, hash it, and map the hash into buckets: perhaps a thousand of them, allocated to arms in the configured proportions.

The properties this buys:

• Stateless and reproducible. Any service, any time, computes the same assignment from the same inputs, no assignment database required.
• Consistent. The user lands in the same arm on every visit, on every device that shares the identifier.
• Independent across experiments. The experiment id inside the hash decorrelates assignments, so being in treatment for experiment A says nothing about arm membership in experiment B, which is what allows hundreds of experiments to overlap on the same traffic.

Gotcha: the salt is load-bearing. Reusing an old experiment's salt reproduces its assignment exactly, delivering a treatment group pre-marinated in a previous treatment's effects. Salts are minted fresh per experiment, and the discipline is boring precisely until the day it was skipped.

A serious platform runs many experiments simultaneously on the same users, and the machinery for that is worth knowing because its failure mode is subtle.

The standard structure is layers. Traffic is divided into independent layers, each owning one aspect of the product: a ranking layer, a UI layer, a pricing layer. Within a layer, experiments split the traffic exclusively, a user sees at most one ranking experiment. Across layers, assignment is independent by construction, different salts, so a user is simultaneously in one experiment per layer, and the cross-experiment combinations average out evenly across arms.

Why this works statistically: experiment B's users are spread evenly over experiment A's arms, so B appears in A's analysis as balanced background noise, raising variance slightly but not bias. The randomisation that protects against the world's confounders protects against other experiments the same way.

• The exception is genuine interaction: two experiments that touch the same surface, one changing button text while another changes button colour, can interact in ways that averaging does not wash out. Same-surface experiments belong in the same layer, where exclusivity prevents collisions, and platforms keep a conflict registry for the cases layers cannot express.
• The practical failure is not statistical but organisational: an unlayered platform where teams hard-code overlapping flags produces users in seventeen experiments with no record of the combinations, and no analysis can untangle what was actually shown to whom.

Assignment says which arm a user would be in. Exposure is the moment they actually receive the experience, and the analysed population must be exposed users, not assigned users.

The difference is dilution. If the experiment changes the checkout page, every user who never reached checkout is assigned but untreated; including them in the analysis mixes millions of identical-by-construction users into both arms, shrinking the measured effect toward zero. A real 4 percent lift among exposed users becomes an insignificant 0.4 percent when diluted ten to one. Trigger the exposure log at the moment the treatment could first affect the user, and analyse those users.

The symmetric trap is conditioning on post-treatment behaviour. If reaching the trigger point is itself affected by the treatment, say the treatment changes which users get to checkout at all, then the exposed populations differ between arms, and the comparison is confounded by the very effect under study. The rule: the trigger condition must be measured identically in both arms and be causally upstream of any treatment effect.

In practice: log the exposure event from the code path that renders the experience, not from the assignment service. Assignment can happen speculatively, in precomputation or prefetching, for users who never see anything; only the rendering path knows the treatment actually reached a human.

Now the guard that watches all of this plumbing. An experiment configured 50/50 should log roughly equal exposed users per arm, roughly, because randomness wobbles. Sample ratio mismatch, SRM, is the statistical check on that expectation: given the configured split and the observed counts, how surprising is the imbalance?

The test is a one-line chi-squared against expected proportions. Its value is entirely in being run always, automatically, before anyone reads a metric.

SRM causes are depressingly mundane, which is why the detector matters more than cleverness. The recurring families:

• Asymmetric loss. The treatment is slower or heavier, so more treatment users bounce, crash, or time out before the exposure event logs. The arm bleeds users in proportion to its own harm, and the survivors are the resilient ones: a double corruption.
• Bots and filtering. Crawler traffic hashes into arms like everyone else, but a bot filter that keys on behaviour the treatment changes filters the arms unequally.
• Redirects. The classic: treatment implemented as a redirect to a new URL. Redirects drop users, get cached, get bookmarked, and every one of those asymmetries lands in the counts.
• Assignment drift. The experiment's traffic allocation was changed mid-flight, or a bugfix altered eligibility, so different eras of the experiment ran different populations, mixed in one dataset.
• Identity fray. Logged-out users, cookie resets and cross-device sessions re-roll their assignment; if the treatment affects login rates even slightly, the re-rolls are asymmetric.

The shared shape: the treatment influences who gets counted, not just what they do. That is also why the fix is never to shrug and analyse anyway, the missing users are systematically unusual, and no reweighting recovers what was never logged.

Diagnosis follows the segments: SRM by browser, by platform, by day, by entry page. The mismatch usually concentrates somewhere, and the concentration names the bug.

Assemble the machinery into the flow a single user takes through a healthy platform, because the order of operations is where correctness lives.

The user arrives; eligibility is evaluated on pre-treatment properties only; the hash of user id, experiment id and salt deterministically selects the arm; the experience renders; and at that rendering moment, the exposure event is logged with the user, experiment, arm and timestamp.

Metrics computation joins exposure logs to outcome events. Before any metric reaches a human, the automated gates run: SRM on the exposed counts, plus data-quality checks, duplicate events, missing days, metric outliers from logging bugs rather than user behaviour.

Only past those gates does statistics begin.

One further institution rounds out the plumbing: the A/A test, an experiment where both arms receive the identical experience. Any systematic difference an A/A shows is by construction a platform bug, biased assignment, asymmetric logging, broken metric joins. Running continuous A/A tests, and demanding they produce significant results at exactly the nominal false positive rate, is how a platform proves its own machinery, and finding an A/A that fails is a gift: it is the cheapest possible notification that every concurrent experiment is suspect.

The plumbing lesson compresses into habits, each purchased by someone's ruined quarter:

1. SRM is a blocking gate, not a dashboard curiosity. A failed check hides the metrics and files a bug; results from an SRM-failed experiment are discarded, never caveated into a deck.
2. Exposure logs at the render path, with the trigger condition identical across arms and causally upstream of treatment effects.
3. Fresh salt per experiment, allocation changes create a new analysis era rather than silently mixing populations, and eligibility rules are frozen at launch.
4. Continuous A/A tests budget the platform's honesty, and their false positive rate is itself monitored.
5. Segment-level SRM, by platform, browser, and day, runs alongside the global check, because a mismatch that cancels in aggregate still poisons the segments where it lives.
6. Bot policy is defined pre-launch and identical across arms, keyed on pre-treatment signals.

None of this involves a p-value, and that is the lesson's point: the majority of untrustworthy results at scale die on these rocks before statistics ever gets a vote. Platforms earn trust through plumbing, and the statistical layer, which has its own famous trap waiting, only matters once the pipeline beneath it is honest. That trap, the experimenter who cannot stop looking at the results, is next.