Tenant isolation and noisy neighbors

Multi-tenant serving usually fails for social reasons before it fails for math reasons. One tenant sends a burst of huge jobs. The engine packs that work into the batch, because packing is what it is built to do. Everyone else's latency rises. Tenant isolation is the set of limits that stops this, so no one tenant can take the whole server.

The big idea

Think of a buffet line where any guest can take as much as they like. Guests who arrive later find the trays empty. A buffet with a per-guest limit keeps the line moving for everyone.

The picture breaks because the engine does not know who is who. It only sees requests. The tenant ID has to be passed to the gateway and the scheduler, or the limits cannot be applied.

Walking through one incident

Tenant A sends 50 concurrent jobs, each with 32k-token prompts
Continuous batching packs A's prefills into every step
Tenant B's interactive chat: TTFT goes from 300 ms to 4 s

The fix combines three limits:

1. A gateway token budget for tenant A. Once its reserved prefill credits run out, new requests wait at the gateway.
2. An engine cap on queued tokens, so no one tenant fills the queue.
3. A separate batch pool for long jobs, so they never share replicas with interactive chat.

Without per-tenant limits, the batcher keeps packing A's work until B's latency target is gone.

The layers of isolation

• Admission caps. Limit in-flight sequences and queued tokens per tenant.
• Fair scheduling. Give each tenant a weighted share of the batch, in the engine or at the gateway.
• Priority classes. Paying tenants get pinned adapters and higher priority.
• Separate pools. Put noisy free-tier traffic on different replicas from enterprise traffic.
• Per-request memory limits. Cap max_tokens and prompt length. One 128k-token prompt should not push everyone else's KV cache out of memory.
• Data isolation. Adapters and KV cache must not leak between requests. Do not log prompts from different tenants into shared debug storage.

Common mistakes

• Expecting continuous batching to be fair. It maximizes throughput. It does not know about fairness.
• One shared queue with no per-tenant caps. A single tenant can fill it.
• Treating data isolation as optional. A cross-tenant leak in an engine is a security bug.

Check yourself

1. Tenant B's latency jumps after tenant A's burst. What limit should you add first? Answer: a per-tenant admission cap on in-flight sequences and queued tokens, enforced at the gateway.
2. Why is an efficient batcher not automatically fair? Answer: it packs whatever work is queued to maximize throughput, so a heavy tenant can crowd out others.

Key takeaways

• Noisy neighbors come from the batcher doing its job on one tenant's burst.
• Stack several limits: admission caps, fair scheduling, priority, pools and per-request memory caps.
• The gateway, the scheduler and the engine each need to see the tenant ID.
• Strict isolation trades utilization for predictability.

Further learning

Read routing and scheduling for how the gateway places requests. For the general pattern outside LLMs, see multi-tenancy.