Token-based rate limits and gateway policy

Request-per-minute limits are the wrong unit when one client sends 100k-token prompts and another sends 20-token classifications. Enforce budgets on estimated input tokens (from the request body) plus max_tokens (or a default cap) as a credit reservation. Reconcile with actual usage when the stream ends.

See rate limiting for token-bucket math. Apply it to tokens, not only to request counts.

Walking through gateway features

Priorities and tiers. Interactive chat and agent tool loops usually outrank offline summarization. Implement priority queues at the gateway or engine: high-tier requests skip ahead when GPU time frees, within fairness caps so a VIP cannot starve everyone forever. Product packaging maps cleanly here: free tier gets lower priority and stricter token budgets; enterprise gets reserved concurrency.

Fair queuing across tenants. Multi-tenant platforms need isolation beyond API keys. Weighted fair queuing by tenant (or by workspace) prevents one noisy customer from absorbing the batch. Combine with per-tenant token budgets and max in-flight caps.

Request cancellation. Clients disconnect constantly — users close tabs, agents cancel tools. Propagate cancellation to the engine so KV blocks and batch slots free immediately. Without this, abandoned streams quietly tax the fleet.

Timeouts and max tokens. Hard ceilings on max_tokens and server-side deadlines stop runaway generations. Prefer explicit application limits over hoping the model emits EOS.

Model routing. Map model names in the OpenAI payload to pools: gpt-foo-fast → 8B FP8, gpt-foo-smart → 70B, embed → embedding servers. Version aliases (foo-prod → pin digest) so clients do not fragment caches across accidental weight changes.

Fallbacks. On 503 or latency SLO breach, degrade to a cheaper model or provider. Log the degradation. Never silently swap a model in regulated flows without policy.

Multi-region. Pin latency-sensitive users to the nearest region with capacity. Use async or batch APIs cross-region. KV-aware routing rarely crosses regions — the cache is not there. Replicate configuration and adapter storage, not live KV, unless you have disaggregated KV infrastructure designed for it.

Track at the gateway: route decisions per replica, estimated vs actual cache hit rate, reject causes, fallback rate, and cancellation rate. If cancellation rate is high and KV stays full, cancels are not reaching the engine.

Common mistakes

• RPM limits only. One 100k-token prompt equals thousands of short classifications.
• Not propagating disconnects. Abandoned streams hold KV until max_tokens.
• Silent model fallback in a regulated flow. Log it, or do not do it.

Check yourself

1. What two numbers should a token budget reserve at admit time? Answer: estimated input tokens from the body, plus max_tokens (or a default cap).
2. Why cancel at the engine, not only at the HTTP edge? Answer: the engine still holds KV and a batch slot until it hears the cancel.

Key takeaways

• Rate-limit tokens, not just requests.
• Reserve credits up front and settle when the stream ends.
• Priorities, fair queues, cancellation and max_tokens belong at the gateway.
• Engine scheduling and gateway policy do different jobs. You need both.

Further learning

Next, shed load on purpose in admission control.