Choosing a model

Score candidates on your task, not on vibes. The only ranking that matters is quality on your eval set, next to latency, price and the constraints you cannot relax.

Work this sequence. Skip a step only if you already know the answer from a prior eval.

Dimensions that actually matter

1. Quality on your eval set
2. Latency — TTFT and tokens per second. Interactive UX often needs TTFT under about 500 ms and a stream that feels continuous. Batch jobs care less
3. Price — $/million input vs $/million output. Output is usually several times more expensive. Cached input can be an order of magnitude cheaper. See tokens and caching
4. Context window — how many tokens of input plus output fit. 128K is common. Longer windows exist, but quality and cost in the long middle still need care
5. Modalities — text-only vs images, PDFs, audio, video. Multimodal tokens are priced and counted differently
6. Tool-use reliability — does the model call the right tool with valid JSON under pressure? Measure this
7. Data residency and compliance — region pinning, zero-retention, HIPAA/BAA, training-on-your-data defaults
8. Licensing — especially for open weights

The useful picture is a Pareto frontier: models that are not dominated on both cost and quality for your metric. Everything below that curve is a bad buy for that task.

The worksheet

1. Prove feasibility on the strongest model. Build the thinnest vertical slice with a frontier or strongest reasoning model. If that fails, the problem is the product, the data or the tools, not the model tier.
2. Freeze an eval set before optimizing. Collect 50 to 200 real examples with expected outputs or graded rubrics. Split easy vs hard. Without this, every model swap is theater.
3. Measure the big model as baseline. Accuracy or judge score, failure modes, TTFT, tokens in/out, $/request. This is the ceiling you are trying to approach cheaper.
4. Step down the ladder. Re-run the same eval on mid and small models, and one open-weight host if relevant. Keep the cheapest model that stays within your quality budget, for example ≤2 absolute points below frontier on critical cases.
5. Slice the traffic. Often the winning design is not one model: a router for easy traffic, a mid model for normal chat and RAG, frontier or reasoning for escalations.
6. Lock IDs and re-test on upgrade. Pin explicit versions. When the vendor releases a new snapshot, re-run evals before flipping production.

Walking through a support summarizer

2M requests/month, about 800 input + 150 output tokens each.

At illustrative mid-tier prices of $3 / $15 per million input/output:

input  = 2e6 × 800 / 1e6 × $3  = $4.80
output = 2e6 × 150 / 1e6 × $15 = $4.50
total  ≈ $9.30 / month

At illustrative frontier prices of $15 / $75:

input  = 2e6 × 800 / 1e6 × $15 = $24
output = 2e6 × 150 / 1e6 × $75 = $22.50
total  ≈ $46.50 / month

If evals show mid-tier within 1% of frontier on summary faithfulness, the frontier model is a $37/month vanity purchase at this volume. The same arithmetic at 200M requests flips the decision toward aggressive optimization and possibly self-hosting. Always plug in current vendor prices. The worksheet matters more than these sample numbers.

The same worksheet on an HR bot is in reasoning models. A design review wants that story, not a leaderboard screenshot.

Common mistakes

• Skipping the frontier proof. You never learn whether the product is feasible.
• No frozen eval. Every swap is theater.
• One model for every lane. Route by failure mode.

Check yourself

1. Mid-tier is $9.30/month, frontier is $46.50, and faithfulness is within 1%. Which do you ship? Mid-tier. The extra $37/month is a vanity purchase at this volume.
2. The strongest model fails the thin slice. Is the mid-tier the problem? No. The product, the data or the tools are the problem.

Key takeaways

• Prove feasibility on a strong model, then step down with a frozen eval.
• Score quality, latency, price, context, tools, residency and license.
• Slice traffic. One model is rarely the cheapest correct design.
• Pin IDs. Re-eval on every upgrade.

Further learning

Why the trophy slide is not the decision: why benchmarks lie.