Serving embeddings and rerankers

Embedding models and most cross-encoder rerankers are encoder-only Transformers: one forward pass over a batch of texts, no autoregressive loop, no growing KV cache across tokens of a response. That flips the bottleneck.

• Work is closer to compute-bound at decent batch sizes (especially with FlashAttention-style kernels and short-to-medium sequences).
• Throughput is measured in texts/s or tokens/s of input, not output tok/s.
• Dynamic batching by sequence length matters. Padding an 8-token query to 512 with a 480-token document wastes huge amounts of FLOPs if you naively pad to max length in a mixed batch.

Walking through indexing math

Suppose you must embed 10 million document chunks averaging 256 tokens, with a model that sustains 8,000 tokens/s of input on one GPU after batching (illustrative).

Total tokens ≈ 10e6 × 256 = 2.56e9 tokens
Time ≈ 2.56e9 / 8,000 ≈ 320,000 seconds ≈ 89 hours on one GPU
On 8 GPUs ≈ 11 hours

That is an offline batch job: use large batches, ignore interactive TTFT, write to object storage or the vector DB in bulk. Online query embedding is the opposite: batch size 1–16, latency target tens of milliseconds, often co-located with the retriever.

Query embedder SLO: p95 < 30 ms. At 5 ms GPU compute for batch=8, you can hide load behind ~2–3 ms of batching wait. Higher wait raises throughput but blows the SLO. Tune the batching window like an admission control parameter, not a constant.

Rerankers score (query, document) pairs. Cost scales with candidates × length. Cap candidates (e.g. 50 → top 10) before the cross-encoder. Otherwise you spend more on rerank than on the LLM that follows.

A typical retrieve → rerank → generate path at 100 QPS:

Query embed:     100 texts/s          → 1× small GPU or CPU ONNX
ANN retrieve:    mostly CPU/RAM
Rerank top-50:   100 × 50 = 5,000 pairs/s → often the hidden GPU hog
LLM generate:    as in chat capacity math

If the reranker is a full cross-encoder without candidate caps, it can cost more than the LLM. Cap candidates, distill a smaller reranker, or use late-interaction models when quality allows. Instrument each stage’s ms and $ separately. “RAG is slow” is not a diagnosis.

Serving stack. Text Embeddings Inference (TEI) is a common high-performance path for embeddings and classification or rerank-style heads. Many teams also put encoders behind NVIDIA Triton with a TensorRT or ONNX backend for multi-model co-location.

Common mistakes

• Padding a mixed-length batch to 512 when p50 length is 64 → about 4–8× wasted FLOPs.
• Sharing one batching window for nightly indexing and online query embed.
• Reranking 200 candidates and then wondering why RAG costs more than chat.

Check yourself

1. How long to embed 10 million 256-token chunks at 8,000 tok/s on one GPU? Answer: about 89 hours; 11 hours on eight GPUs.
2. Why bucket embedding batches by length? Answer: mixed lengths pad to the longest item and burn FLOPs on pad tokens.

Key takeaways

• Encoders are one-shot and often compute-bound. Meter texts/s, not output tok/s.
• Batch by length buckets. Split offline indexing from online query embed.
• Cap rerank candidates. The cross-encoder is often the hidden GPU hog.
• TEI or Triton are the usual serving paths.

Further learning

Text Embeddings Inference (Hugging Face)A blazing fast inference solution for text embeddings models - huggingface/text-embeddings-inferenceGitHub

Next, speech serving has a different clock: audio duration, not tokens.