Serving embeddings, rerankers, speech and diffusion

Chat LLMs dominate the discourse, but production AI platforms spend a surprising fraction of GPU time on everything else: embedding millions of chunks for RAG, reranking candidates, transcribing audio, synthesizing speech, generating images, and scoring tabular features with a GBDT. Those models do not share the prefill/decode KV story. Treat them like chat and you will over-provision the wrong resource, batch the wrong way, and miss SLOs that were easy on paper.

One gateway, many model types is fine. One autoscaling signal for all is not. Scale embedding Deployments on batch queue depth, diffusion on GPU util or queue time, LLMs on KV and waiting queue as in autoscaling, speech on audio-seconds queued, classic ML on p99 score latency.

Inventory every model on the critical path of a RAG+chat or voice product: query embedder, document embedder (batch), reranker, LLM, maybe ASR/TTS. Give each its own Deployment (or Triton model) with an appropriate batcher and SLO. Benchmark with realistic length mixes — synthetic all-128-token batches lie about padding waste.

For embedding product design see embeddings. For generative image and video theory see generative models. For voice UX see voice agents.

Batching by model type

Model type	Batch on	Limit by	Metric
Chat LLM	Tokens / sequences (continuous)	KV + SLO	tok/s, TTFT, TPOT
Embedding / rerank	Texts bucketed by length	Padding waste, VRAM	texts/s, p95 ms
ASR	Audio clips by duration	Mel memory, encoder size	RTF, hours/day
TTS	Utterances / frames	First-audio SLO	ms to first audio
Diffusion	Prompts (often small B)	Step count × VRAM	images/min, e2e s
GBDT / small ONNX	Rows if any	p99 ms	scorings/s

Teams often ask whether embeddings can share the LLM’s H100s “when idle.” Sometimes yes for nightly indexing. Rarely yes for online query embedding next to chat decode. LLM decode wants large KV arenas. Embedding batches want contiguous free memory for big padded tensors. They fight. A chat spike starves the embedder and blows RAG p95. Share only for offline jobs with preemption. Give online embedders, rerankers and TTS their own smaller GPUs (L4/A10) or CPU pools.

Concepts in this section

Step	Concept	What it answers
1	Embeddings and rerankers	How do you batch encoders and cap rerank cost?
2	Speech serving	Batch ASR vs streaming TTS — which clock?
3	Diffusion serving	Why is latency steps times step time?
4	Classic ML serving	Why do GBDTs belong on CPU?

Tag every metric with model_name and model_type. Without that, on-call pages “GPU high” with no idea which of five colocated models is guilty.

Common mistakes

Padding a mixed-length embedding batch to 512 when p50 length is 64 → about 4–8× wasted FLOPs.
Running Whisper large on CPU for "cost savings" until the backlog grows faster than realtime.
Serving SDXL at 50 steps on an interactive button with no distillation.
Putting XGBoost on a GPU "for consistency" and paying more latency than a local CPU library.
Autoscaling diffusion on CPU util while GPUs are saturated.

Key takeaways

Encoders and rerankers are one-shot and often compute-bound. Batch by length buckets. Meter texts/s.
Indexing millions of chunks is an offline throughput problem. Query embedding is a latency problem. Split them.
Speech splits into batch ASR (efficiency) vs streaming TTS/ASR (first audio / partials).
Diffusion cost is step count × step time. Distillation beats micro-optimizing a 50-step loop.
Classic ML belongs on CPU ONNX or Triton FIL, not on an H100 chat pool.
Match batching policy and scale signals to the model type.