Input vs output token cost

Providers and honest self-host models price input and output differently because the work differs. Prefill is fat, often compute-bound, and scales with prompt length. Decode is skinny, memory-bound, and serial. Cached input is mostly a memory read of existing KV.

If you collapse them into one "tokens" number, you cannot tell whether to shrink prompts, raise cache hit rate, or speed up decode.

The big idea

A restaurant that prices soup and steak the same will mis-price every menu. Input tokens are not output tokens. Cached input is closer to a refill than a new dish.

The analogy breaks for agents: they resend huge histories, so input can dominate the bill even though output is "what the user reads."

Walking through the three meters

A prefill-heavy workload (agents with huge contexts, RAG dumping 50k tokens of docs, tool loops that resend history) can make input the majority of spend. A decode-heavy chat (short prompts, long answers or chain-of-thought) is dominated by output tok/s and KV growth.

An agent averages 12,000 input tokens (tools + history) and 400 output tokens per turn, 2 turns per user action, at 50 actions/s:

Output tok/s      = 50 × 2 × 400     = 40,000
Fresh input tok/s = 50 × 2 × 12,000  = 1,200,000   (before cache)

If prefix caching hits 80% of input tokens, fresh prefill drops to 240k tok/s equivalent — often the difference between needing a prefill fleet and fitting on the decode pool. Price the product on cached vs fresh lines or you will undercharge every agent feature.

Common mistakes

• One "tokens" counter. You cannot tell whether to cache, shorten prompts, or speed decode.
• Ignoring cached vs fresh. An 80% prefix hit can cut prefill by 5×.
• Assuming output is always the bill. Prefill-heavy agents flip that.

Check yourself

1. 50 actions/s, 2 turns, 400 output tokens each. What is output tok/s? Answer: 50 × 2 × 400 = 40,000.
2. Same agent sends 12,000 input tokens per turn and the cache hits 80%. What is fresh input tok/s? Answer: 50 × 2 × 12,000 × 0.20 = 240,000.

Key takeaways

• Meter fresh input, cached input and output separately.
• Prefill-heavy and decode-heavy products need different fixes.
• Prefix cache hit rate is a cost lever, not only a latency lever.

Further learning

Next, decide API vs self-host.