Guardrails and vanity metrics

Optimizing one number creates incentive to game it. Guardrail metrics are the things you refuse to sacrifice. Counter-metrics are the specific ways the primary metric usually breaks. Vanity metrics move easily and feel good without changing outcomes: raw downloads, registered users, pageviews, total emails sent.

In A/B tests, declare primary + guardrails before launch. A treatment that lifts conversion 2% but spikes refunds 8% is not a win — it is a future support fire.

The big idea

Actionable metrics change what you build next week and connect to value or revenue. Absolute counts answer "are we bigger?" Rates answer "are we healthier?" Per-user averages hide power users; pair with medians or percentiles when skew is high (revenue, messages, API calls).

A classic trap: celebrating MAU while DAU/MAU falls — you acquired tourists who never return. See retention. "We grew signups 40% after the conference" is not a product win until those signups activate and retain at rates comparable to organic.

Teams game what you measure. Defense:

1. Qualify the event — "projects with ≥3 tasks completed," not "projects created."
2. Use counter-metrics — empty-project rate, spam-report rate.
3. Prefer hard-to-fake value — nights booked and stayed beats nights favorited.
4. Audit outliers — top 1% of accounts often reveal abuse or instrumentation bugs.

Denominators matter as much as numerators. "Activation rate" with a bloated signup denominator from a viral spam campaign looks terrible even if activated users are fine. Always ask: what entered the denominator this week that did not last week?

Walking through it

North star (collaborative docs/week) rose 12% over eight weeks. Paid conversion flat. Expansion revenue flat.

Two compatible stories: (1) you delivered more value to free users without a path to pay — packaging problem; (2) the metric inflated via single-player "docs" that are not collaborative — definition drift. Check the fraction of WCED that involve ≥2 editors:

week_1: multi_editor_share = 0.62
week_8: multi_editor_share = 0.41

The star rose while the quality of the star fell. Fix the definition (require ≥2 editors) before celebrating growth. Metric hygiene is product strategy.

When a PM proposes a new input metric, ask: (1) Can we move it in six weeks with product work? (2) If it doubles and the north star does not move, what did we learn? (3) What guardrail stops us from gaming it?

Common mistakes

• Output theater. Shipping velocity as the team metric. You get more tickets, not more value.
• No veto list. If you would not block a ship for it, it is not a guardrail.
• Celebrating a signup spike without activation and retention of that cohort.

Check yourself

1. Conversion rises 2% and refunds rise 8%. Is the experiment a win?

No. The guardrail failed. It is a future support fire, not a ship.

2. WCED rose 12% while the multi-editor share fell from 0.62 to 0.41. What happened?

Definition drift: the star inflated with single-player docs. Tighten the event before celebrating.

Key takeaways

• Pair the primary metric with 2–4 guardrails you will actually veto a ship for.
• Prefer actionable outcome counts over vanity stock metrics; use ratios as diagnostics.
• Qualify events, watch denominators and audit outliers — teams game what you measure.

Further learning

a16z's catalog of startup metrics is a useful checklist of what investors and operators actually track.

16 Startup Metrics (a16z)We have the privilege of meeting with thousands of entrepreneurs every year, and in the course of those discussions are presented with all kinds of numbers, measures, and metrics that illustrate the promise and health…Andreessen Horowitz

Next: map the journey with AARRR.