Guide

Metrics and experimentation

CUPED

Subtract the predictable part of a metric using pre-experiment data. You get the same MDE with less traffic — if the experiment was valid.

74 of 128 · 2 min read

CUPED (Controlled-experiment Using Pre-Experiment Data; Deng et al., 2013) reduces metric variance by adjusting for a pre-period covariate — often the same metric before the experiment.

Users who spent a lot last week tend to spend a lot this week. Subtracting the predictable part shrinks noise. Bing reported on the order of ~50% variance reduction for some metrics — equivalent to needing roughly half the sample for the same power when the reduction holds.

CUPED does not fix SRM, peeking or interference. It makes a valid experiment more sensitive.

Concept

Because the covariate X is measured before assignment, it is independent of treatment under correct randomization. The adjustment does not bias the treatment effect. It only removes variance.

Walking through the adjustment

Y_cuped = Y − θ · (X − mean(X))
θ ≈ Cov(Y, X) / Var(X)

Y is the metric in the experiment. X is the same (or a related) metric before assignment. θ is the slope you'd get from regressing Y on X.

Without CUPED, you needed n = 28,800 total for δ = 0.01 at p = 0.10. If CUPED cuts variance in half, required n scales with variance:

n_new ≈ 28,800 × 0.5 = 14,400 total

Same MDE in half the calendar time (or half the traffic). That is why mature platforms invest in CUPED — not because the stats are prettier, but because experiment capacity is the scarce resource.

Deepen into sequential testing and CUPED only after the org stops peeking and fishing. Sensitivity on a polluted test just makes you confidently wrong faster.

Common mistakes

  • Using CUPED to rescue SRM or peeking. It cannot.
  • Covariates measured after assignment. Then X can be affected by treatment and you bias the effect.
  • Expecting 50% every time. Bing's figure was for some metrics. Re-measure θ on yours.

Check yourself

You peeked for a week, then apply CUPED. Is the readout trustworthy?

No. CUPED shrinks variance. It does not un-peek a test.

Why must X be measured before assignment?

If X is after assignment, treatment can change X, and the adjustment biases the effect.

Key takeaways

  • CUPED subtracts pre-experiment predictability so you need fewer users for the same MDE.
  • X must be pre-assignment. The adjustment is unbiased under correct randomization.
  • Sensitivity is not validity. Fix peeking, fishing and SRM first.

Further learning

Improving the Sensitivity of Online Controlled Experiments by Utilizing Pre-Experiment Data (Deng et al.)2013-02CUPEDImprovingSensitivityOfControlledExperiments.pdfai.stanford.edu

Back to the map in experimentation pitfalls.

Experimentation pitfalls

Peeking, metric fishing, sample ratio mismatch, novelty effects, network interference, CUPED and Twyman's law — how trustworthy experiments go wrong and how to catch them.

Sample size for A/B tests

How many users you need, why tiny lifts demand huge traffic, and how to pick an MDE you would actually ship on.

A/B testing and experimentation

Designing randomized experiments, sizing samples with real arithmetic, reading lift and confidence intervals, and shipping only what the data actually supports.