Token limits for LLM APIs

For most APIs, one request costs about the same as another. LLM APIs break that assumption. A request with a 200-token prompt and a 50-token answer is cheap. A request with a 150,000-token document and a 4,000-token answer costs a GPU far more. A limit of "60 requests per minute" says little about load, so LLM providers limit on several dimensions at once.

The big idea

Use a request limit to cap per-request overhead, a token limit to cap work, and a concurrency limit to cap streams. Streaming requests hold a slot for many seconds, so a concurrency cap matters too.

The hard part is that output length is unknown when the request starts. The usual answer is to reserve the worst case up front and refund what was not used. Think of a restaurant that holds a table for the largest party it might seat, then frees the chairs that stay empty.

Walking through it

Suppose the limit is 100,000 tokens per minute, enforced as a token bucket that refills at about 1,667 tokens per second:

Request arrives: 12,000 input tokens (counted with the tokenizer), max_tokens = 4,000
Reserve up front: 12,000 + 4,000 = 16,000 tokens   → bucket must hold at least 16,000
Response finishes having generated 900 tokens
Refund:           4,000 − 900 = 3,100 tokens back into the bucket
Net charge:       12,900 tokens

Reserving the maximum protects the GPUs. The refund keeps the limit fair, because a client is charged for what it actually used.

This has a side effect. Setting max_tokens far above what you need can get you throttled sooner, because the reservation counts against your budget until the refund.

Providers publish these limits in different ways. Anthropic documents separate input and output token limits per minute, enforced with a token bucket that refills continuously. OpenAI publishes requests per minute and tokens per minute for each model and usage tier. Read the provider's docs for the exact rules.

If you build an internal LLM gateway, you will implement this pattern yourself, often with budgets per team on top. How the serving side schedules admitted requests onto GPUs is covered in routing and scheduling. The product-level design is in the LLM chat product case study.

Common mistakes

• Counting only requests. Ten huge prompts can cost more than a thousand small ones.
• Forgetting the refund. If reserved tokens are never returned, heavy users are throttled by their own estimates.
• Setting max_tokens much higher than needed. The unused reservation can trigger throttling.

Check yourself

1. A request reserves 16,000 tokens. The response uses 900 of its 4,000 max_tokens. What is the net charge?

   Answer: 12,900 tokens, the 12,000 input tokens plus the 900 generated.

2. Why reserve max_tokens up front instead of charging the actual output afterward?

   Answer: the output length is unknown at the start, so reserving the maximum protects capacity, and the refund returns the unused part.

Key takeaways

• LLM requests differ in cost by orders of magnitude, so limit tokens as well as requests.
• Output length is unknown at the start, so reserve max_tokens and refund the unused part.
• Streaming holds a slot for seconds, so cap concurrent requests too.
• A large max_tokens can trigger throttling even when the response is short.

Further learning

This is the last concept in the rate-limiting series. Return to the rate limiting overview for the full map.