LLM rate limits and overload

LLM traffic is expensive and bursty. One request asking for a 20-page answer can use more GPU time than a hundred short questions. Limit by tokens, not requests. When the fleet is full, shed load by tier and degrade in steps. A flat "try again later" for every user is the worst option.

The big idea

Each user or organization gets a token budget per window, such as tokens per minute and tokens per day. Admission checks the budget with an estimate of the request's input and maximum output tokens. The actual usage is charged when the response finishes, which corrects the estimate.

Overload is a separate problem from quotas. A user can be under quota while the whole fleet is saturated. Admission must also look at fleet-level queue depth and wait time.

Walking through it

Degrade in steps, so the least painful option comes first:

1. Shorten the context. Use a smaller summary and fewer retrieved passages.
2. Route to a smaller model. Lower quality, but the request still succeeds.
3. Cap output length. Stop very long answers from holding GPUs.
4. Queue with a time limit. Free and batch tiers wait, and paid tiers go first.
5. Reject with a clear message. Return a retry-after value, so clients back off.

A queue with a time limit makes this concrete. Say a free tier gets a 10-second wait and the paid tier gets 2 seconds:

GPU fleet at 95% of peak, queue depth 40

paid request    enters queue, first in line, wait ~1.5 s   -> served
free request    enters queue, wait limit 10 s               -> served once a slot opens
free request    waited 10 s, still queued                   -> 429 with retry-after 15 s

Tier priority means an enterprise customer's request can jump a batch job. Batch and background work should always be the first to yield.

Multi-tenant inference needs fairness as well. One large customer must not take all the GPU slots. Give each tenant a share, and reject or queue a tenant's excess before it reaches the shared queue.

Be honest about the trade-off. A smaller model under overload gives a worse answer, and some users won't like that. Tell them which model answered, and let the user retry on the larger model when capacity returns.

Common mistakes

• Rate limiting by request count. A single long request can starve everyone else.
• Failing all requests at once. Step-wise degradation keeps the product usable.
• Ignoring queue wait in admission. The fleet can look fine on quota and still be overloaded.
• No retry-after value. Clients retry immediately, and the overload gets worse.

Check yourself

1. Why limit by tokens rather than requests?

   Token counts track GPU cost. One huge request can use more capacity than hundreds of small ones.

2. The fleet is at 95% of peak. What should the free tier see first?

   A longer queue wait with a time limit, or a smaller model. Paid requests go first, and clients get a retry-after when they're rejected.

Key takeaways

• Limit by tokens per window, and charge the actual usage when the response ends.
• Admission should look at fleet queue depth as well as per-user quota.
• Degrade in steps: shorter context, smaller model, capped output, queue, then reject.
• Give every rejection a retry-after value, and tell users when a smaller model answered.

Further learning

Next: tools and untrusted input, which covers the other source of cost and risk.

For how the serving layer handles batching and queues, see the inference serving platform.