Evals as product specs

If you cannot define good, you cannot ship responsibly. An eval set — fixed inputs plus a scoring method — is how quality becomes a requirement instead of a vibe. Model swaps and prompt tweaks become engineering changes measured against a bar you wrote down.

Product managers who insist on a quality number without a dataset are asking for theater. Engineers who refuse any quality bar are asking for an incident.

Walking through a support-draft bar

Illustrative bars by use case:

Support-draft feature. Offline set: 200 real tickets (PII scrubbed) with gold "should cite policy §N" labels. Bar: ≥90% citation faithfulness, ≤5% critical policy violations on the set, p95 draft latency ≤8 s. Auto-send stays off until those hold on three consecutive model or prompt candidates.

Online: sample 2% of drafts for human review weekly; page if the critical-violation rate exceeds 1%. The PRD's "AI quality" section is these numbers — not "feels good in the demo."

Evals also force scope honesty. If you cannot assemble a labeled set for a claimed use case, you probably do not understand it well enough to sell it. Start from production traces and failure taxonomies, not from a vendor's generic "helpfulness" score.

Keep artifacts as the source of truth and invest in harnesses that survive model swaps. Roadmaps that say "wait for the next model" are not strategies. Ship the learning loop now so a better model is a week of measurement, not a rebuild.

Common mistakes

• No dataset, just a number. "90% quality" without a set is theater.
• Generic helpfulness scores. They do not match the job you sell.
• Enabling auto-send because the demo looked good. The bar is three consecutive candidates, not one stage pass.

Check yourself

The PRD says "the model should feel accurate." What do you replace that with?

A frozen set, a scorer, thresholds (e.g. ≥90% faithfulness, ≤5% critical violations) and an online guardrail.

You have no labeled tickets for the use case you want to sell. What does that tell you?

You do not understand the use case well enough yet. Start from traces, not a vendor score.

Key takeaways

• Write eval bars into the PRD. They are the quality spec.
• Offline set plus online guardrail. Auto-send waits on both.
• A missing labeled set is a scope problem, not an engineering delay.

Further learning

How to build the harness: evals. Next, make the economics work in AI product economics.