The LLM landscape and model choice

Picking a model is the highest-leverage decision in an LLM product after you invent the use case. The wrong choice burns money on a frontier model you do not need, or ships a small model that silently fails on the hard 10% of traffic. The landscape changes every few months. The process for choosing does not.

This is a map of what exists as of when this was written, and a worksheet for deciding, not a ranking that will still be true next quarter.

Quality is not linear with price. On many extraction tasks a mid or small model matches a frontier model once the prompt and schema are tight. On multi-step tool use and messy instructions, the gap is still large.

Concepts in this section

Planning for churn

Model names, context limits and prices churn constantly. Engineer for it.

• Thin provider adapters. Keep prompts, tools and business logic behind an interface that takes messages in and returns text | tool_calls. Libraries like LiteLLM or the Vercel AI SDK help. Do not let vendor SDKs sprawl into every file. See LLM APIs.
• Config-pinned model IDs. The id lives in env or config, not hardcoded in ten services.
• Eval suite in CI. A model bump is a pull request that must pass the suite.
• Cost and quality dashboards. Online metrics catch silent regressions when a "drop-in replacement" is not.

The team that can swap models in an afternoon with confidence owns a durable advantage. The team that rewrote half the product around one vendor's quirks does not.

When finance asks for a 40% cost cut, do not start with "switch vendors." Start with shorter outputs, prompt caching, traffic routing to smaller models, and batch for offline jobs. Model swaps are the last lever after you know quality headroom from the worksheet.

Common mistakes

• A beauty contest without a task. Chat preference and schema adherence are weakly correlated. Eval on the actual output contract.
• Optimizing before measuring. Jumping to the cheapest model in week one freezes a mediocre baseline. Spend frontier tokens to discover the ceiling, then economize.
• One model for every lane. A reasoning model for "rewrite this subject line" wastes tokens. A tiny model for multi-tool planning wastes incidents.
• Ignoring tool-call reliability. Beautiful prose with invalid tool JSON at 5% will dominate on-call.
• License surprise. An open-weight checkpoint that forbids your SaaS use case, or an API that trains on prompts by default, is a legal problem discovered too late.
• Alias drift. Pinning latest means a silent quality change on someone else's release day.

Key takeaways

• Frontier APIs maximize quality and speed-to-ship. Open weights maximize control and sometimes unit cost after ops.
• "Open weights" is not open source. Read the license.
• Reasoning models buy hard-problem accuracy with expensive thinking tokens. Route them.
• Leaderboards shortlist. Your eval set decides.
• Start strong to prove the product, then step down the price-latency ladder with frozen metrics.
• Abstract providers, pin versions and re-eval on every release.

Further learning

Artificial AnalysisComparison and analysis of AI models and API hosting providers. Independent benchmarks across key performance metrics including quality, price, output speed & latency.artificialanalysis.ai LMArena leaderboardlmarena.ai AI Engineering (Chip Huyen)oreilly.com

Karpathy's intro if you need the mental model of what an LLM is before choosing one: