Guide

Metrics and experimentation

Peeking at experiments

Checking the p-value daily and shipping when it dips below 0.05 inflates false positives from 5% into the teens or worse.

69 of 128 · 2 min read

An A/B test with broken methodology is worse than no test: it lends false confidence to a bad ship. The most common break is peeking — repeatedly checking a p-value and stopping when it dips below 0.05.

The temptation is rational: "traffic is expensive; why wait two weeks if Friday looks conclusive?" The math is not. Each interim look is another chance for noise to look like signal.

Concept

Optional stopping (peeking): the p-value is calibrated for one pre-planned look, not a sightseeing tour. Under the null, a fixed-horizon test falsely rejects about 5% of the time. With continuous peeking, the false positive rate can climb toward 20%+ depending on how often you look.

Walking through the inflation

Suppose under H₀ you check a two-sided p-value once at the end: P(false ship) ≈ 5%.

If instead you examine the test independently on 10 days and stop at the first p < 0.05, a rough Bonferroni-style bound says the family-wise error could approach:

1 − (1 − 0.05)^10 ≈ 1 − 0.95^10 ≈ 1 − 0.5987 ≈ 40%

That bound is conservative (daily tests are correlated). Industry simulations of naive peeking often land in the teens to twenties of percent false positive rate — several times the nominal 5%. You did not "get lucky early"; you bought Type I errors on an installment plan.

Cost: ten "winning" UI tweaks ship; six are noise; two hurt retention you only notice next quarter.

Fixes:

  1. Fixed horizon — compute required n, set end date, do not decide early. You may monitor for severe regressions or SRM, but not for optional winning.
  2. Sequential testing — group sequential, always-valid inference, or SPRT spend α across looks so overall Type I error stays controlled. Use a library; do not invent thresholds.
  3. Bayesian with pre-registered decision rules — continuous monitoring is fine if the threshold was set in advance. "Looks good to me today" is still peeking.

When leadership asks to "just peek and pull the launch forward," offer a sequential design approved up front, or a hard regression monitor without optional winning.

Common mistakes

  • Monitoring the primary for a win. Watching crashes is not peeking. Shipping on Friday's p-value is.
  • Inventing sequential thresholds. Use a stats library or platform.
  • Changing to one-sided after seeing a dip. That is p-hacking, not a design choice.

Check yourself

You sized for two weeks. Friday of week 1 is p = 0.04. Do you ship?

No. The p-value is for one planned look. Wait for the horizon, or use a pre-registered sequential method.

Is it peeking to page on a crash-rate guardrail mid-test?

No. Severe regression monitors are allowed. Optional stopping on the primary is not.

Key takeaways

  • Peeking inflates false positives from ~5% into the teens or worse.
  • Fix with a locked end date, or a sequential method that spends α on purpose.
  • Monitor for harm and SRM. Do not optional-stop on a win.

Further learning

How Not To Run an A/B Test (Evan Miller)Are you falling victim to the peeking problem? Learn how to design and run an experiment properly.evanmiller.org

Next, the other way clean tests die: metric fishing.

Experimentation pitfalls

Peeking, metric fishing, sample ratio mismatch, novelty effects, network interference, CUPED and Twyman's law — how trustworthy experiments go wrong and how to catch them.

A/B testing and experimentation

Designing randomized experiments, sizing samples with real arithmetic, reading lift and confidence intervals, and shipping only what the data actually supports.

Metric fishing

Test 20 metrics and something will be "significant." Ship on one pre-registered primary, and treat the rest as leads.