Guide

Metrics and experimentation

Metric fishing

Test 20 metrics and something will be "significant." Ship on one pre-registered primary, and treat the rest as leads.

70 of 128 · 2 min read

Testing one primary metric at α = 0.05 is calibrated. Testing 20 metrics and shipping if any is significant is a fishing expedition.

Slices multiply the same problem. Five countries × two platforms is ten more chances to get lucky. The same discipline applies to LLM eval suites — see LLM as judge.

Concept

Multiple comparisons: under independence, the chance of at least one false positive among k tests is 1 − (1 − α)^k. For ship decisions, prefer one pre-registered primary. Put the rest on a guardrail or secondary list.

Walking through a metric menu

α = 0.05, k = 20 independent null metrics:

P(≥1 false positive) = 1 − (0.95)^20 ≈ 1 − 0.3585 ≈ 64%

Nearly two-thirds of such experiments will show something "significant" even when nothing works.

Bonferroni: use α' = α / k. For k = 20, α' = 0.0025. Harsh but simple; protects family-wise error.

FDR (Benjamini–Hochberg): controls the expected fraction of false discoveries among the rejected set — less conservative when many metrics are truly null. Fine for exploration dashboards. For ship decisions, still prefer one pre-registered primary.

Trade-off

Product sense wants "also check revenue, retention, latency…" Put those on a guardrail / secondary list with pre-declared interpretation: secondaries can generate follow-up hypotheses, not first-pass ship rights.

Even a clean overall lift can hide harm in a segment: new users up, power users down. Looking at segments is necessary — and dangerous — because of multiplicity. Pre-declare a small set of slice dimensions you will always check (platform, new vs returning). Treat other slices as exploratory.

Overall conversion: control 10.0%, treatment 10.8%. Split by tenure:

New users (60%):     6.0% → 7.5%   (lift +1.5 pp)
Returning (40%):    16.0% → 15.0%  (lift −1.0 pp)
overall_control = 0.6·6.0 + 0.4·16.0 = 10.0%
overall_treat   = 0.6·7.5 + 0.4·15.0 = 10.5%

Shipping a change that trains new users well but annoys returning users may raise short-term conversion and hurt retention. Options: target only new users; iterate on returning; or kill if returning revenue dominates. The pitfall is reporting only the aggregate headline.

Common mistakes

  • Shipping because a secondary went green. That was exploration, not the decision metric.
  • Slicing after the fact until a story appears. Pre-declare a few slices.
  • Bonferroni on the primary after seeing the data. Correction is a design choice, not a rescue.

Check yourself

Twenty null metrics, α = 0.05. About how often will something look significant?

About 64% of the time, if the tests are independent.

Overall lift is +0.8 pp. Returning users are −1.0 pp. What do you report?

Both. The headline hides harm. Decide with the pre-declared slices, not the aggregate alone.

Key takeaways

  • One pre-registered primary for ship rights. Secondaries are leads.
  • 1 − (1 − α)^k is why a metric menu lies.
  • Pre-declare a few slices. Treat the rest as exploration.

Further learning

Next, a data-quality alarm that beats any green primary: sample ratio mismatch.

Experimentation pitfalls

Peeking, metric fishing, sample ratio mismatch, novelty effects, network interference, CUPED and Twyman's law — how trustworthy experiments go wrong and how to catch them.

Peeking at experiments

Checking the p-value daily and shipping when it dips below 0.05 inflates false positives from 5% into the teens or worse.

LLM-as-a-judge

Using models to grade models: when a judge is the right grader, how to design one, and why an uncalibrated judge is a vibes function.