Reducing LLM tokens

Tokens are both latency and money. Prefill (reading the prompt) often dominates TTFT on long contexts. Decode (writing the answer) dominates total wall time and usually costs more per token than input.

Cut tokens before exotic serving tricks. Structured outputs and retrieval beat giant prompts.

Three cuts

Shorten the prompt. Drop unused tools from the schema. Summarize history instead of appending every turn forever. Prefer retrieval of the top few chunks over stuffing an entire help center into the window. See context engineering.

Cap output length. Set max_tokens to what the product needs. Ask for bullets when prose wastes tokens. Prefer structured outputs when the UI only needs fields. JSON with short keys often beats a long essay you parse afterward.

Collapse chain-of-thought when you do not need it. Reasoning models that emit long internal traces can be right and ruinously slow. Use them when quality requires it. Otherwise a smaller non-reasoning model with a crisp prompt is cheaper.

A "short answer" prompt without max_tokens still lets a reasoning model emit thousands of hidden or visible tokens. Cap generations per surface. For agents, cap total tokens per session and tool iterations.

Walking through a support turn

Support bot prompt: 4,000 input tokens of static policy plus 200 tokens of user question, generating 350 tokens of answer. At roughly $3 / M input and $15 / M output (mid-tier frontier pricing, order of magnitude as of when this was written):

input   = 4,200 / 1e6 × $3  = $0.0126
output  =   350 / 1e6 × $15 = $0.00525
total   ≈ $0.018 per turn

Cut static context to 1,200 tokens via RAG and cap answers at 150 tokens:

input   = 1,400 / 1e6 × $3  = $0.0042
output  =   150 / 1e6 × $15 = $0.00225
total   ≈ $0.0065 per turn    (about 64% cheaper, before caching)

Same product, fewer tokens, same quality bar on the eval. Caching the remaining static prefix is the next lever, in prompt caching.

Common mistakes

• Leaving unused tools in the schema. The model still reads them. Prefill grows for no benefit.
• Appending the full thread forever. Summarize or window the history.
• No output cap. Reasoning traces will surprise you.

Check yourself

1. Input 4,200 tokens at $3/M and output 350 at $15/M. About how much is one turn? About $0.018. Input $0.0126 plus output $0.00525.
2. You cut static policy from 4,000 tokens to 1,200 with RAG. What else should you cap? Output length. Here, 150 tokens instead of 350.

Key takeaways

• Prefill dominates TTFT on long prompts. Decode dominates wall time and output spend.
• Drop unused tools, retrieve instead of stuffing, cap max_tokens.
• Skip long thinking traces unless evals show you need them.
• Measure the cut against the same eval suite.

Further learning

Next, send easy traffic to a cheaper model in routing and cascades.