Shipping and specialties
When AI fits
Pick jobs with high value, tolerable errors and a verifier. Skip demos that cannot be checked.
AI product strategy starts with which jobs tolerate a probabilistic system. LLMs are strong when the task has messy language or perception in, and structured or checkable work out. They are weak when you need perfect recall of private facts without retrieval, or zero-tolerance correctness with no verifier.
Treat the model like a powerful, unreliable junior employee: hire it for the right tasks, measure its work, and put guardrails on irreversible actions.
Concept
A verifier is anything that can check the model's output without needing another unverifiable judgment — unit tests for code, JSON schema validation, citation overlap, human approval, or a domain rule engine. AI products without verifiers are demos. AI products with verifiers can set SLAs.
The big idea
A practical filter:
| Criterion | Prefer AI when… | Avoid / constrain when… |
|---|---|---|
| Value of success | Minutes saved, deals unblocked, tickets deflected | Novelty demo with no workflow owner |
| Cost of error | User can review; undo exists; stakes moderate | Irreversible money, medical or legal action without a human gate |
| Verifiability | Tests, schemas, citations, graders exist | "Sounds good" is the only check |
| Frequency | Happens often enough to learn and amortize UX | One-off that never builds a feedback loop |
A smoke alarm is useful because a false beep is cheap and a missed fire is not. Flip that and you have a bad AI job: auto-issuing refunds from email sentiment. The analogy breaks when the "beep" itself is expensive (a bad customer email, a wrong medical note) — then you need a human gate, not just a model.
Walking through two jobs
Good fit: "Draft a reply from this ticket + policy docs; agent sends only after approve." Error cost is bounded. The verifier is the human click. Value is handle time.
Poor fit: "Auto-issue refunds from email sentiment alone." Error cost is cash. No reliable verifier. Regulators and customers will not accept plausible mistakes.
Say no when a deterministic rules engine or search already solves 90% of the job; when the demo requires a human silently fixing outputs before the customer sees them; when you have no path to a verifier and the stakes are high; or when the only differentiation is the underlying model brand, which a competitor can swap tomorrow.
AI can still appear later as an assist layer on a solid classical product. Nail the system of record, then accelerate the workflow. That sequencing often beats "AI-first" skimming of a domain you do not own.
Common mistakes
- Starting from the model. "We have GPT access" is not a use case.
- Skipping the verifier. If you cannot assemble a labeled set, you do not understand the job well enough to sell it.
- Selling theater. A human fixing outputs offstage is not a product.
Check yourself
You want to auto-send support replies with no human in the loop. What must exist first?
A verifier and a quality bar — for example citation faithfulness on a labeled set — plus a kill switch.
Search already finds the right policy 90% of the time. Should you replace it with an LLM?
Probably not. Add AI as an assist on the remaining mess, or leave the deterministic path.
Key takeaways
- Choose AI jobs with high value, tolerable errors and a verifier.
- Cost of error decides whether the model drafts, suggests or is blocked.
- Classical software first is often the right sequence; AI later as an accelerator.
Further learning
Next, pick the product shape in copilot vs agent.
Related
AI product strategy
What to build with AI, how to defend it, how to price it, and how evals become product requirements.
Copilot vs agent vs automation
Positioning is a product choice. Assist, act with tools, or run in the background — each implies different UX and liability.
Evals as product specs
If you cannot define good, you cannot ship. An eval set is how quality becomes a requirement instead of a vibe.