Guide

Production and case studies

Model routing and cascades

Send easy work to a small model and hard work to a frontier model. A cascade tries cheap first and escalates when a check fails.

124 of 155 · 2 min read

Not every request needs the strongest model. A router classifies the task, or uses heuristics, and sends easy work to a small, fast model and hard work to a frontier model.

A cascade is the sequential version: try the cheap model first; if confidence is low or a verifier fails, escalate.

Concept

Before you start: you need a detectable easy/hard split on your eval suite. Until that split is real, one well-chosen model is simpler. See choosing a model.

How a cascade runs

1. Small model answers with a confidence score or a "needs_escalation" flag
2. If high confidence and code checks pass → return
3. Else call the frontier model, optionally with the small model's draft as context

Routing signals that work in practice:

  • Intent or ticket category from a classifier
  • Presence of account-mutating tools (refunds always escalate)
  • Retrieval score floor (weak retrieval → stronger model or a human)
  • Code-based checks failing on structured output

Trade-off

Cascades save money when most traffic is easy, often 70 to 90% of support and FAQ. They add latency on the hard path, because that path pays for two model calls. Keep routing logic versioned and evaluated. A bad router that sends refunds to a tiny model is a safety bug, not a cost win.

If the small model always escalates, you pay for two calls on every request. Log escalate rate. If it sits above about 40%, the router is wrong or the small model is under-powered for the task mix.

Walking through the bill

From the support-bot cost model: after RAG and caching, mid-tier spend was about $14,220 per month. Route 75% of turns to a small model at about $0.40 / M input and $1.60 / M output, same token shapes, and 25% stay on the mid-tier.

Small (75%):  ≈ $1,260 input + $648 output = $1,908
Mid   (25%):  ≈ $3,555
Total         ≈ $5,500 / month     (−88% vs the $45,000 baseline)
Cost / convo  ≈ $0.0055

The quality gate is the point. If tool-use pass rate drops 12 points on the routed slice, you do not ship the savings.

Common mistakes

  • Routing before you can detect easy vs hard. You will send refunds to the tiny model.
  • Ignoring escalate rate. Above ~40%, you are paying for two calls and still using the expensive model.
  • Unversioned routers. Treat routing rules like a prompt: version, eval, then ship.

Check yourself

  1. 80% of questions are FAQ and 20% need tools. Router or one frontier model? A router, if a classifier or heuristic can tell them apart on evals.
  2. Escalate rate is 55%. What is going wrong? The small model is too weak or the router is sending hard work down. You pay twice.

Key takeaways

  • Route when evals show a detectable easy/hard split. Otherwise use one good model.
  • Cascades add latency on the hard path. Worth it when most traffic is easy.
  • Refunds and other mutations always escalate.
  • Log escalate rate. Above about 40%, the cascade is not saving money.

Further learning

Back to the cost model. Offline jobs belong on batch LLM APIs, not on a cascade.

Latency and cost optimization

Streaming, routing, cascades, batching and parallelism: making LLM apps fast and affordable without dropping quality.

The LLM landscape and model choice

Frontier vs open-weight models, reasoning models, and a process for picking one with your own evals.

Evaluating LLM apps

A fixed set of inputs plus a way to score outputs, so you can tell whether a change helped.