Production and economics
Serving speech models
ASR cost scales with audio duration. Batch similar clip lengths. Streaming TTS is judged on time-to-first-audio, not realtime factor alone.
Speech models do not share the prefill/decode KV story. Input size scales with audio duration, not tokens in a prompt box. GPU memory is dominated by activations for the current audio batch, not a giant chat KV. Treat batch file transcription and a live voice agent as different products.
Concept
Realtime factor (RTF) is how long the model takes relative to the audio. RTF 1.0 means one second of audio takes one second to transcribe. Throughput is often hours of audio per GPU-day. Conversational TTS is judged on time-to-first-audio, the delay until the user hears sound.
Walking through Whisper-style ASR
OpenAI Whisper and descendants are encoder–decoder sequence models: audio → mel features → encoder → autoregressive text decoder. Properties that matter for serving:
- Chunk long audio (30 s windows are a common pattern) for parallelism and memory.
- Batch multiple clips of similar duration. Mixed 3 s and 30 s clips pad badly.
- Streaming ASR (partial transcripts) needs chunked or streaming-native architectures. Vanilla full-utterance Whisper is offline-ish unless you window cleverly.
Running Whisper large on CPU for "cost savings" is a common trap: the backlog grows faster than realtime.
Walking through streaming TTS
Text-to-speech for voice agents is latency-critical. Users hear time-to-first-audio, not just RTF.
- Stream tokens from the LLM into a TTS that emits audio frames early.
- Prefer models and runtimes that support chunked synthesis. Avoid waiting for the full sentence when the product is conversational.
- Co-locate TTS with the orchestrator region to avoid shipping PCM across continents.
Trade-off
Batch ASR vs streaming ASR. Batch (file upload transcription) maximizes GPU efficiency with large batches. Streaming voice agents optimize time-to-first-partial and barge-in; expect lower utilization. Do not put both on one Deployment with a single batching policy.
Export metrics that match the physics: realtime factor, queue seconds of audio, error rate by language for ASR; time-to-first-byte of audio and underruns for TTS. Tag every metric with model_name and model_type.
The big idea
A call center that scores agents on "calls per hour" will rush every conversation. A call center that scores "seconds until the customer hears a greeting" will staff the greeting differently from overnight voicemail dump. Batch ASR is the voicemail dump. Streaming TTS is the greeting. The picture breaks if you force one GPU pool and one batcher to do both.
Common mistakes
- Mixing 3 s and 30 s clips in one ASR batch and paying pad cost on the short ones.
- Waiting for a full sentence before TTS starts, then blaming the LLM for voice lag.
- One Deployment for file transcription and the voice-agent path.
Check yourself
- What should you batch ASR clips by? Answer: similar duration, so padding waste stays low.
- Which number does a voice-agent user feel first? Answer: time-to-first-audio, not hours of audio per GPU-day.
Key takeaways
- ASR cost scales with audio duration. Chunk long files. Batch similar lengths.
- Batch ASR maximizes hours/day. Streaming ASR/TTS maximizes first audio and partials.
- Do not share one batching policy across those two jobs.
- Measure RTF and queue-seconds for ASR, time-to-first-audio for TTS.
Further learning
Practical Whisper pipeline context:
Next, diffusion serving is a step-count problem.
Related
Serving embeddings, rerankers, speech and diffusion
Inference beyond chat LLMs. Encoders, speech, diffusion and classic ML each have a different bottleneck and a different batcher.
Voice agents
Real-time speech AI: STT→LLM→TTS pipelines vs speech-to-speech models, latency budgets and turn-taking.
Serving embeddings and rerankers
Encoder-only models do one forward, no growing KV. Batch by sequence-length buckets. Indexing is a throughput job. Query embed is a latency job.