Guide

Metrics and experimentation

Sample ratio mismatch

If you asked for 50/50 and observe 48/52 on a large sample, stop. The experiment is invalid until you explain the split.

72 of 128 · 2 min read

If you requested a 50/50 split and observe 48/52 on a large sample, something is wrong with assignment, exposure logging, bot filtering or triggering. Sample ratio mismatch (SRM) is a data-quality alarm: do not interpret metric lifts until it is explained.

A green primary metric beside a red SRM is not a ship signal. DoorDash and other experimentation teams treat SRM as a first-class automatic check for this reason.

Concept

SRM means the observed traffic split does not match the requested ratio by more than telemetry noise. A χ² goodness-of-fit test against the expected ratio is standard. With huge n, even tiny imbalances are "significant" — what matters is whether the gap is larger than noise and whether it correlates with a segment.

Walking through the arithmetic

Expected 50/50. Observed: control 82,000, treatment 78,000. Total N = 160,000.

expected each = 80,000
χ² = (82000 − 80000)² / 80000 + (78000 − 80000)² / 80000
   = (2000)² / 80000 + (−2000)² / 80000
   = 4,000,000 / 80,000 × 2
   = 50 + 50
   = 100

On 1 degree of freedom, χ² = 100 is astronomical (p ≪ 10⁻²⁰). Stop.

Common causes: treatment crashes and users never log exposure; CDN caches one variant; logged-out vs logged-in bucketing differs; a filter applied after randomization; bots dropped in one arm only.

Asymmetric bot dropping creates SRM-like bias. Filter bots with the same rules in both arms. Triggered experiments (randomize only users who open a modal) estimate effects on the triggered population — do not multiply that lift by all MAU. State the denominator: "among users who viewed pricing, +1.0 pp paid."

SDE2s are often the ones who notice "treatment has 8% fewer exposures on Safari." That observation is more valuable than a fancy CUPED implementation.

Common mistakes

  • "Analyzing through" SRM. A skewed split means the comparison is broken.
  • Treating any χ² significance as SRM on huge n. Check size and segment correlation, not just p.
  • Different bot or login filters per arm. You manufactured the mismatch.

Check yourself

Requested 50/50. Observed 82k vs 78k on 160k users. χ² = 100. Ship if the primary is green?

No. Stop. Explain the split before you read any lift.

Treatment crashes on Safari so those users never log exposure. What do you see?

SRM — fewer treatment exposures — and a biased metric among the survivors.

Key takeaways

  • SRM means the experiment is invalid until explained.
  • Run a χ² check automatically. A green primary + red SRM is not a ship.
  • Look for crashes, caches, login bucketing and asymmetric filters.

Further learning

Addressing the Challenges of Sample Ratio Mismatch in A/B Testing (DoorDash)careersatdoordash.com

Next, effects that fade or leak across users: novelty and interference.

Experimentation pitfalls

Peeking, metric fishing, sample ratio mismatch, novelty effects, network interference, CUPED and Twyman's law — how trustworthy experiments go wrong and how to catch them.

A/B testing and experimentation

Designing randomized experiments, sizing samples with real arithmetic, reading lift and confidence intervals, and shipping only what the data actually supports.

Designing an experiment

Write the hypothesis, primary metric, guardrails, unit and exposure definition before anyone looks at a p-value.