Interactive and batch pools

A serving platform that mixes interactive chat and offline summarization on one pool will miss the chat SLO the first time a batch job grabs KV. Split the work. Interactive replicas hold a reserved floor. Batch consumes spare capacity, or a cheaper queue, and can be killed.

A typical mix: ~50% interactive chat (1–4k input, 200–800 output, strict SLO), ~30% RAG/agent (4–32k input, cacheable prefixes), ~20% batch/offline (mixed, soft latency, preemptible).

Walking through the platform pieces

Interactive pool. 70B-class FP8, modest tensor parallel (TP=2 on a node is common), continuous batching, prefix caching, multi-LoRA on a shared base. Cap concurrent LoRAs in GPU RAM. Spill cold adapters to CPU or NVMe. Route by tenant_id → adapter_id at the gateway. See multi-LoRA.

Gateway. Auth, per-tenant token quotas and $ budgets, model and LoRA routing, prefix-aware routing, priority lanes (interactive > batch). Retries with idempotency keys where safe. Never blindly duplicate side-effecting tool calls. See routing.

Autoscaling. Scale on waiting queue, KV utilization and TTFT, not GPU util. minReplicas sized for cold-start × ramp (weights are 100 GB-class). Warm pool of ready replicas for evening peaks. Separate HPA for batch workers. See autoscaling.

Batch tier. Run offline jobs on spare interactive capacity with preemption (kill batch mid-request when KV pressure rises), or on spot / older GPUs with a queue (Kafka/SQS) and no TTFT SLO. This is how you raise utilization toward the optimistic $/M without missing chat SLOs.

70B in FP16 needs ~140 GB weights alone. FP8 roughly halves weight memory, frees KV headroom, and is the default starting point for 70B-class interactive serving as of when this was written. Validate with evals. Keep a BF16 canary path for quality cliffs.

Pipeline parallel is usually wrong for tight TPOT SLOs unless model size forces it. Expert parallel only if you actually serve an MoE.

The big idea

A hospital keeps ICU beds reserved and puts elective surgery on a list that can slip. Chat is the ICU. Batch is elective. The picture breaks if "reserved" still means one shared KV arena with no preemption: then the elective case still evicts the emergency.

Common mistakes

• One HPA on GPU util for chat and batch. GPU looks busy while TTFT burns.
• No preemption. A 100k-token offline job pins the interactive replica.
• Floating latest tags for LoRAs, so a tenant silently swaps weights.

Check yourself

1. What three signals should scale the interactive pool? Answer: waiting queue, KV utilization and TTFT — not GPU util.
2. How does batch raise utilization without stealing chat? Answer: run it on spare or spot and preempt it when KV pressure rises.

Key takeaways

• Split interactive and batch. Preempt batch first.
• Scale interactive on queue, KV and TTFT. Keep a warm pool for cold starts.
• FP8 plus modest TP plus prefix cache is the usual 70B interactive stack.
• Start colocated. Disaggregate only when prefill fights decode.

Further learning

Next, drill node loss, spikes and bad rollouts.