Guide

LLM fundamentals

Reasoning models

Extra inference-time compute, often a hidden chain of thought, billed as output tokens. Use them on the hard slice, not on "summarize this email."

3 of 155 · 2 min read

Reasoning models spend extra compute at inference time, often emitting a hidden or visible chain of thought, before answering. Providers bill that extra thinking as output tokens, or as a separate thinking budget.

They shine on math, coding, multi-hop analysis and planning. They are wasteful on "summarize this email" and "extract the invoice total."

Concept

Before you start: ordinary sampling already lets a model write a scratchpad if you ask. Reasoning models budget extra tokens for that scratchpad by default, and you pay for them even when the task is easy.

When the extra tokens pay off

Use a reasoning model when error analysis shows reasoning failures: a wrong plan, a missed constraint, brittle multi-step logic. Prefer a fast non-reasoning model when failures are knowledge or format failures. Fix those with RAG, tools or structured outputs first.

Thinking tokens are expensive. Reserve them for the hard slice of traffic via routing.

Trade-off

You buy hard-problem accuracy with output spend and latency. A subject-line rewrite does not need that. A multi-tool plan often does.

Walking through a mix

An internal "ask HR" bot over policy PDFs. A mid-tier non-reasoning model with RAG scores 90% on the suite. A small model scores 78% and invents policy numbers. About 8% of questions are tagged hard: benefits math, conflicting policy versions.

easy 92% of traffic  →  mid-tier, no thinking budget
hard  8% of traffic  →  reasoning model
blended quality      ≈ 91%

All-frontier-reasoning would be higher quality on the easy slice by a point you cannot feel, and much more expensive. The reasoning model is a lane, not a default.

Common mistakes

  • Defaulting every call to a reasoning model. Subject lines and extractions do not need thinking tokens.
  • Using thinking to paper over missing retrieval. If the chunk is wrong, more thought will not invent the right policy number reliably.
  • No max_tokens or thinking cap. A "short answer" still emits thousands of hidden tokens.

Check yourself

  1. Failures are "invented policy numbers" and the right paragraph is in the docs. Reasoning model or RAG? RAG first. That is a knowledge failure, not a reasoning failure.
  2. Why route only 8% of HR questions to a reasoning model? Thinking tokens are expensive. The hard slice is benefits math and conflicts. The rest does not need them.

Key takeaways

  • Reasoning models buy extra inference-time compute, billed as output.
  • Use them for plans, math, code and multi-hop analysis.
  • Fix knowledge and format failures with RAG, tools and schemas first.
  • Route them. Do not default them.

Further learning

Next, the worksheet that decides which tier you actually ship: choosing a model.

The LLM landscape and model choice

Frontier vs open-weight models, reasoning models, and a process for picking one with your own evals.

Choosing a model

Prove the product on a strong model, freeze an eval, then step down the ladder. Your suite decides, not a leaderboard.

Model routing and cascades

Send easy work to a small model and hard work to a frontier model. A cascade tries cheap first and escalates when a check fails.