Voice latency budgets

Humans notice awkward gaps. Natural turn-taking often needs roughly 500–800 ms from the moment the user stops speaking to the first audio of the reply. Exact comfort varies by culture and context. Miss that window and the product feels broken even if the transcript is perfect.

Text chat forgives multi-second thinking. Voice does not. That single constraint reshapes model choice (favor low time-to-first-token), prompt size (favor short system prompts and cached prefixes), retrieval (pre-fetch where you can) and even product scope (defer hard tool chains with a spoken handoff to async SMS).

Concept

Before you start: a cascaded stack adds STT, LLM and TTS latencies. Serial addition blows the budget. Overlap is how you get it back.

Walking through 700 ms

Target: ~700 ms voice-to-voice for a web agent (ambitious but realistic with streaming). A naive serial cascade blows it:

Endpointing wait          400 ms
STT final                 300 ms
LLM time-to-first-token   400 ms
TTS time-to-first-byte    200 ms
Network / jitter buffer   100 ms
--------------------------------
Serial total             1400 ms   (too slow)

Streaming and overlap reclaim the budget:

Endpointing (semantic)             200 ms
STT partials already flowing         0 ms  (hidden)
LLM TTFT on speculative partial    250 ms
TTS TTFB on first sentence         120 ms
Net / buffer                        80 ms
--------------------------------
Overlapped path                   ~650 ms

Example

Twilio's public guidance for cascaded voice agents targets on the order of ~350 ms STT, ~375 ms LLM TTFT, ~100 ms TTS TTFB as component goals — before network. Treat vendor numbers as starting points. Measure your p50/p95 mouth-to-ear on real calls.

Levers that matter more than framework choice: colocating STT/LLM/TTS in one region, smaller/faster models for the conversational tier, streaming everywhere, and not waiting for "perfect" finals before starting the LLM. A single cross-coast hop per stage quietly eats the 700 ms budget.

Example

A support line's p95 time-to-first-audio was 1.8 s. Breaking it down: 700 ms endpointing silence, 400 ms waiting for STT final, 500 ms LLM TTFT on a giant system prompt, 200 ms TTS. Fixes: semantic endpointing (~250 ms), start LLM on stable partials, shrink system prompt and enable prompt caching, stream TTS. p95 fell to ~750 ms. Containment rate rose because callers stopped hanging up in the gap.

Transport adds its own tax

Path	Typical use	Notes
WebRTC	Browser / mobile apps	Low latency, handles NAT, built for media
WebSockets	Server-to-server audio	Simpler ops sometimes; watch buffering
SIP / telephony	Phone numbers, call centers	μ-law 8 kHz; PSTN latency adds up

Generate TTS at telephony sample rates when on PSTN so you are not paying for resample artifacts and extra delay. Keep STT, LLM and TTS in the same cloud region as the media edge when you can. Realtime constraints from realtime systems apply: do not buffer entire utterances server-side; measure p95; design for packet loss.

Common mistakes

Waiting for a perfect STT final before starting the LLM.
A giant uncached system prompt on the conversational path.
Crossing coasts between media, STT, LLM and TTS.

Check yourself

Why is 400+300+400+200+100 = 1,400 ms the wrong way to add a cascade? The stages can overlap. Partials, speculative LLM work and streamed TTS hide most of that wait.
p95 first audio is 1.8 s and 700 ms is silence endpointing. What do you change first? Endpointing. Semantic end-of-turn cut that slice to ~250 ms in the example.

Key takeaways

Aim for roughly sub-second voice-to-voice. Overlap streams; do not add stages serially.
Measure mouth-to-ear on real calls, not vendor component slides.
Colocate media and models. Generate TTS at the transport's sample rate.
Shrink and cache the system prompt. Start the LLM on stable partials.