Admission control and load shedding

When offered load exceeds goodput, something must give. Prefer an explicit policy over silent TTFT blowups. Options, from kindest to bluntest:

1. Queue with a budget. Allow waiting up to e.g. 2 s for interactive. If expected wait exceeds budget, fail fast.
2. Shed lowest priority. Drop or defer batch and offline first.
3. Reject new prefills, finish decodes. Decodes already hold KV. Starving them wastes sunk cost. Many systems prefer to stop admitting heavy prefills first.
4. Return 429/503 with Retry-After. Tell well-behaved clients to back off. Pair with client jitter.
5. Fallback. Route to a smaller model, another region, or a hosted API — only if product allows the quality and cost trade.

Publish queue time as a first-class SLO alongside TTFT. Users experience queue time as TTFT. If you only measure in-engine prefill, you will miss gateway meltdown.

The big idea

A packed restaurant does not seat a walk-in by squeezing them onto a table that is already eating. It quotes a wait, turns people away, or offers the bar menu. The picture breaks if the host's wait estimate is a lie: then everyone stands in the lobby and the kitchen still melts. Predicted start time has to be honest.

Walking through a simple policy

Admit if predicted start time keeps the TTFT SLO. Else 429 with Retry-After. Shed batch first and reclaim KV. Optionally degrade non-premium tenants to a smaller model for the duration of the incident. Disable expensive features (long RAG, high-temperature sampling, a speculative draft that increases load) if you still need room.

Publish the policy in the runbook. Product and support should know that 429s under overload are intentional safety valves, not "the GPUs being flaky."

The ecosystem is converging on patterns instead of every company writing a one-off Envoy filter. Gateway API Inference Extension extends Kubernetes Gateway API so an Inference Gateway can call an Endpoint Picker (EPP) that selects backends using model-server metrics — not only kube endpoint readiness. Features include model-aware routes, priorities, LoRA-aware targeting, and pluggable pickers.

llm-d and related projects assemble Kubernetes-native distributed inference: routing components, well-known metrics contracts, and integration toward disaggregated serving. Treat them as references for which signals to expose from your engines even if you are not ready to adopt the whole stack.

Minimum practical path if you are not on those projects yet:

1. Sidecar or engine /metrics with KV utilization, waiting queue, prefix hit rate.
2. A picker that uses affinity_bonus - queue_penalty - kv_pressure.
3. Hard admission limits per pool.
4. Dashboards that show hit rate and reject rate next to GPU duty cycle.

Common mistakes

• Parking overflow on the fullest GPU instead of rejecting. That is how you OOM.
• Measuring only in-engine TTFT. Gateway queue time is what users feel.
• Retry storms without jitter after a 503.

Check yourself

1. Why finish in-flight decodes before admitting new prefills? Answer: decodes already hold KV; killing them wastes sunk cost and still does not free the prompt work.
2. What should a client do with a 429 that carries Retry-After? Answer: wait that long, add jitter, then retry; do not immediately stampede.

Key takeaways

• Admit with a queue-time budget. Shed on purpose.
• Prefer rejecting prefills over starving decodes.
• Queue time belongs on the SLO dashboard next to TTFT.
• Standardized inference gateways (Gateway API Inference Extension, llm-d) encode these signals as the fleet grows.

Further learning

llm-dAchieve state of the art inference performance with modern accelerators on Kubernetes - llm-d/llm-dGitHub Gateway API Inference ExtensionIntroduction - Kubernetes Gateway API Inference Extensiongateway-api-inference-extension.sigs.k8s.io vLLM metrics designdocs.vllm.ai