Designing an experiment

A clean readout starts before the first user is assigned. If you decide the metric after seeing the chart, you are no longer testing a hypothesis — you are hunting for a story.

Write the design down first. Then lock it. The statistics in reading results only mean what this page said they would mean.

Walking through a brief

Write these six things before looking at results:

1. Hypothesis — "Showing social proof on the pricing page increases signup→paid conversion because trust, not confusion, is the blocker."
2. Primary metric — one number you will use to decide (paid conversion within 7 days of exposure). Tie it to a north star or input.
3. Guardrails — refund rate, latency, crash rate, support tickets — metrics that can veto a ship.
4. Randomization unit — usually user (or account) so the same person does not see both variants. Session-level randomization is simpler and wrong when learning spans sessions.
5. Exposure logging — record when the user became eligible to see the treatment.
6. Population — new users only? One country? Exclude internal employees and bots.

Feature flags (config and environments) are the usual delivery mechanism: the flag both serves the variant and stamps exposure.

Assignment must be deterministic given user_id and experiment_salt (usually a hash modulo). Sticky assignment prevents flicker. Changing the salt mid-test invalidates causal claims — treat salt changes like schema migrations.

When not to test, and A/A first

Skip an A/B test for irreversible migrations with no meaningful counterfactual, legal compliance, or bugs where the correct behavior is unambiguous. Run an A/A test (both arms identical) first when the stack, metric pipeline, or randomization unit is new — you should see about α false positives over many trials, and SRM should stay quiet.

Most product experiments should be two-sided: you care if treatment is better or worse. One-sided tests give a bit more power in one direction and blind you to harm. If someone proposes one-sided after seeing the data, that is p-hacking.

Share a one-page brief in the PR that flips the flag: hypothesis, primary metric SQL, n target, end date.

Common mistakes

• Picking the primary metric after the readout. That is metric fishing. See experimentation pitfalls.
• Logging assignment instead of exposure. "Assigned" ≠ "saw the modal" ≠ "completed the flow." Dilution shrinks real lifts.
• Re-bucketing mid-test. A salt or percentage change mid-flight breaks the comparison.

Check yourself

The flag assigned 50,000 users to treatment. Only 12,000 opened the modal the flag was meant to change. Which number is your sample?

The 12,000 who were exposed. The rest never saw the treatment and will dilute the lift.

A teammate wants a one-sided test after seeing treatment slightly down. What do you say?

No. The sidedness was a design choice. Changing it after the data is p-hacking.

Key takeaways

• Pre-register hypothesis, primary metric, guardrails, unit and exposure.
• Randomize by user when the experience spans sessions.
• Deliver through feature flags with deterministic, sticky assignment and honest exposure logs.

Further learning

Next, size the test in sample size for A/B tests.