Reading A/B test results

The readout is a decision, not a trophy. You report the numbers you pre-registered, then ask whether the lift is both real and worth the cost.

A p-value is not "probability the treatment works." Under the null (no difference), it is the probability of seeing a result at least as extreme as yours. A 95% confidence interval is the range of plausible true lifts. A CI that excludes 0 matches a two-sided test at α = 0.05 under standard assumptions.

Walking through a readout

Control: 14,400 users, 1,440 conversions → p_c = 10.00%. Treatment: 14,400 users, 1,584 conversions → p_t = 11.00%.

absolute lift = 0.1100 − 0.1000 = +0.0100  (+1.0 pp)
relative lift = 0.0100 / 0.1000 = +10%

Approximate standard error for the difference of two proportions:

SE ≈ sqrt( p_c(1−p_c)/n_c + p_t(1−p_t)/n_t )
   = sqrt( 0.10·0.90/14400 + 0.11·0.89/14400 )
   = sqrt( 0.00000625 + 0.000006797 )
   = sqrt( 0.000013047 ) ≈ 0.00361

Rough 95% CI ≈ lift ± 1.96·SE:

0.0100 ± 1.96·0.00361
0.0100 ± 0.00708
→ [0.0029, 0.0171]

The CI excludes 0, so the result is statistically significant at α = 0.05. At 100k eligible users/month:

extra_payments/month ≈ 100,000 × 0.01 = 1,000

If average first payment is $40, that is ~$40k/month gross — before refunds and COGS. Stats say "real." Finance and product sense say "ship or iterate."

Report, in order: sample sizes and exposure counts (watch for SRM); observed lift (absolute and relative); 95% CI; p-value for the pre-registered primary only; guardrail deltas.

Layers and holdouts

Running many concurrent tests on the same users can create interactions. Mature orgs use overlapping experiment layers with orthogonal bucketing, or reserve a global holdout — a few percent of users who see the old world for months — to measure the cumulative effect of the whole program on the north star.

Know which layer your flag sits in. When in doubt, document co-running experiments on the analysis ticket.

Common mistakes

• Shipping on p < 0.05 alone. Check the CI width, guardrails and cost.
• Reading every secondary metric as a ship right. Secondaries generate follow-ups. See metric fishing.
• Stopping early because Friday looked green. That is peeking.

Check yourself

The 95% CI for lift is [−0.002, +0.018]. Is the test a ship?

Not on stats alone. Zero is inside the interval, so you have not rejected no-effect. Decide with the pre-registered rule, or run longer.

A result is significant at +0.1 pp of conversion on a change that adds a fragile dependency. What else do you need?

Practical significance: is the lift worth the cost, and do guardrails hold?

Key takeaways

• Report lift, CI, p-value on the primary, and guardrails — in that spirit, not p alone.
• Statistical significance is not a ship ticket. Pre-commit a lower bound and cost test.
• Watch SRM, layers and peeking before you trust the interval.

Further learning

The definitive industry book on online experiments.

Next, how clean tests die in production: experimentation pitfalls.