Randomization and causality

Observational data tells you what happened. It rarely tells you why. A randomized experiment isolates cause by making the treatment the only systematic difference between groups.

Most ideas fail. Teams at large consumer companies routinely report that roughly two-thirds of experiments do not move the primary metric in the intended direction. That is why you test instead of shipping on a hunch.

The big idea

A checkout redesign launches on Monday. Conversion rises 3%. Was it the redesign, a holiday, a competitor outage, or a marketing campaign that also started Monday? You cannot know. Power users adopt a feature first and their retention looks great — selection did that, not the feature.

An A/B test holds the world fixed in expectation: same week, same traffic mix, same seasonality, split across arms. Correlation from a dashboard can fool you. Randomization is the cleanest causal instrument most product teams will ever have.

Picture a fair coin. You are not matching users one-by-one. You are trusting chance so that, on average, the groups are twins. The picture breaks when assignment is not actually random — a bug, a cache, or a bot filter that quietly sorts people.

Walking through it

Two stories, same chart, different causes.

Ship and watch. You launch a new banner. Users who click it convert more. Those users were already more motivated. The banner selected them; it did not create the conversion.

Randomize. Each user is hashed into control or treatment. Both groups contain motivated and unmotivated people in the same mix. If treatment converts 11% and control converts 10%, the extra point is the banner — plus noise, which sample size and a confidence interval will quantify.

For the statistics underneath (random variables, distributions), see probability and information.

Common mistakes

• Reading a launch chart as causation. Anything else that moved that week is a confounder.
• Session-level assignment when habit spans sessions. The same person sees both variants and the comparison is polluted.
• Analyzing assignment, not exposure. Users who never became eligible to see the treatment dilute the effect.

Check yourself

A pricing page conversion jumps 4% the week you ship a redesign and a holiday sale. Can you credit the redesign?

No. Sale and redesign are confounded. Only a randomized split isolates the page.

Why does "users who used the new feature retained better" fail as proof the feature works?

Users who find and use a feature are already more engaged. That is selection, not a treatment effect.

Key takeaways

• Dashboards show what happened. Randomization is how you learn what caused it.
• Most tested ideas do not move the primary metric. That is expected.
• The unit of randomization should match how people experience the product — usually the user, not the session.

Further learning

Next, write the experiment down before you look at results in designing an experiment.