Metrics and experimentation
Novelty and interference
Day-one spikes fade, and my treatment can change your outcome. Short tests and independent-user math both lie here.
Two assumptions break quietly. First, that a two-day lift will last. Second, that treating me does not change your outcome. Novelty (and primacy) break the first. Interference breaks the second.
Shipping on a 48-hour "winner" for a homepage redesign is how you optimize for surprise, not for value. If the effect is real and large, a full week will still show it.
Concept
Novelty: users interact more with anything new for a few days, then revert. Primacy: some users prefer the familiar control and adapt slowly. Interference (a SUTVA break): my treatment changes your outcome — invites you receive, wait times you see, inventory you can buy.
Walking through a decaying spike
Examine day-by-day lift curves. A spike on day 1–2 that decays to zero is novelty. Mitigations: minimum 1–2 week duration covering weekly cycles; long-term holdouts — keep a small % on the old experience for months to measure durable effect on retention and revenue.
Users learn. A new navigation may look negative in week 1 (primacy) and positive in week 4. A dark-pattern treatment may look positive until trust erodes. Carryover after the experiment ends (client-side state) can pollute the next test — washout periods or fresh cohorts help.
Marketplace learned equilibrium (drivers change hours based on a fee experiment) may not show in a one-week window. When the system adapts, short tests measure the wrong horizon.
When units interfere
Standard A/B tests assume units do not interfere. That fails when:
- Social graphs — I see a new sharing UI; you receive more invites.
- Marketplaces — treating some riders changes wait times for control riders.
- Shared resources — caches, inventory, sales quotas.
| Design | Idea | Use when |
|---|---|---|
| Cluster randomize | Assign whole teams / cities / graph communities | Strong within-cluster interaction |
| Switchback | Alternate treatment over time in the same region | Marketplace geo effects |
| Ego-cluster / network experiments | Explicit graph designs | Social products |
These need more traffic and specialist design. If interference is mild, acknowledge bias direction qualitatively rather than pretending the p-value is gospel.
Common mistakes
- Calling a 48-hour UI spike a win. Wait for the novelty curve to flatten.
- Ignoring marketplace leakage. Control riders are not untreated if you moved the treated ones.
- Skipping washout. Last week's client-side state is this week's confounder.
Check yourself
Lift is +12% on day 1, +2% on day 3, 0% on day 10. What is that?
Novelty. Do not ship on the day-1 number. Extend or use a long-term holdout.
You treat 10% of riders in a city. Control wait times fall. Is the test valid?
Not as a simple user-level A/B. Treated riders changed the marketplace for everyone. Consider switchback or cluster designs.
Key takeaways
- Novelty and primacy distort short UI tests. Prefer week-scale runs and holdouts.
- Interference breaks independent-user math. Use cluster or switchback when needed.
- Watch day-by-day curves and learned-equilibrium horizons.
Further learning
Next, buy sensitivity without buying validity: CUPED.
Related
Experimentation pitfalls
Peeking, metric fishing, sample ratio mismatch, novelty effects, network interference, CUPED and Twyman's law — how trustworthy experiments go wrong and how to catch them.
A/B testing and experimentation
Designing randomized experiments, sizing samples with real arithmetic, reading lift and confidence intervals, and shipping only what the data actually supports.
Retention and engagement
Retention curves, cohort analysis, DAU/MAU and why retention is the foundation of growth. Start here for the map.