Token pricing

Vendors quote USD per million tokens, with separate rates for fresh input, cached input and output. Exact multiples change by model and year — read the current price card. The shape of the bill does not.

The shape of the bill

cost ≈ (in_tokens × in_price + out_tokens × out_price
        + cached_tokens × cached_price) / 1_000_000

Output length is the lever most teams underuse. Asking for "a concise answer in ≤3 bullets" and setting max_tokens often saves more money than switching models.

Walking through a reply-suggestion feature

Feature: AI reply suggestions. 500,000 calls/day. Average 1,200 input tokens (800 of which are a stable system+docs prefix eligible for caching) and 80 output tokens.

Illustrative prices: input $3/M, cached input $0.30/M, output $15/M.

fresh_input  = 500_000 × 400 / 1e6 × $3     = $0.60
cached_input = 500_000 × 800 / 1e6 × $0.30  = $0.12
output       = 500_000 × 80  / 1e6 × $15    = $0.60
daily total  ≈ $1.32
monthly      ≈ $40

Without caching, the same input would cost 500_000 × 1,200 / 1e6 × $3 = $1.80 per day on input alone ($0.60 → $1.80). Caching did not change the model. It changed the bill by keeping the big prefix identical across calls.

Scale the same feature to 50M calls/day and you are in five-figure monthly territory even with a mid-tier model. That is when latency and cost work — routing, smaller models, shorter outputs — stops being optional.

Another brief: help-article search

Product asks: "AI search over our 5,000 help articles, ~100k queries/day."

Retrieval: 8 chunks × 400 tokens = 3,200
System + tools                     = 1,000
Query + template                     = 200
Output answer average                = 250
Fresh input per query                = 4,400

At the same $3 / $15 per M tokens:

input/day  = 100_000 × 4_400 / 1e6 × $3  = $1.32
output/day = 100_000 × 250   / 1e6 × $15 = $0.375
month      ≈ 30 × $1.70 ≈ $51

If 2,500 of those input tokens are a cacheable "always the same" policy+tool prefix, cached rates can cut a large fraction of input cost — but only if the prefix bytes are identical. A unique timestamp in the system prompt means the cache never hits.

Embeddings and vector DB costs are separate line items. Do not forget them in the same brief. The point of the exercise is to make token arithmetic a required artifact next to the mock.

Common mistakes

• Ignoring output price. At 3–5× input, a chatty answer dominates the bill.
• Breaking the cache with a timestamp or request ID at the front of the prompt.
• Forecasting with words, not tokens. Count with the model's tokenizer.

Check yourself

1. In the reply-suggestion example, what does caching save on input per day? $1.08. Uncached input is $1.80; cached plus fresh is $0.72.
2. Why can a timestamp in the system prompt erase that saving? The prefix bytes change every call, so the cache never hits.

Key takeaways

• Cost is (tokens × price) / 1,000,000 per class: fresh input, cached input, output.
• Output is often 3–5× input. Short answers are a cost control.
• Prompt caching rewards a stable prefix. Unique bytes at the front destroy it.
• Put a monthly token forecast with arithmetic in the design review.

Further learning

LLM caching and latency and cost. Next, how to budget the window so history does not eat the savings.