Voice agents

A voice agent converses with speech instead of, or in addition to, text. The hard problem is staying inside a human conversational latency budget while handling interruptions, noisy audio, telephony quirks and the same tool-use and safety issues as text agents.

Humans notice awkward gaps. Natural turn-taking often needs roughly 500–800 ms from the end of the user's speech to the first audio of the reply. Miss that window and the product feels broken even if the transcript is perfect. Text chat forgives multi-second thinking. Voice does not.

That constraint reshapes model choice (favor low time-to-first-token), prompt size (short system prompts and cached prefixes), retrieval (pre-fetch) and product scope (defer hard tool chains to async SMS). Everything below is in service of sounding present without giving up control.

Concepts in this section

Read them in order. Each one builds on the one before.

Step	Concept	What it answers
1	Cascaded vs speech-to-speech	Control and evals, or lower latency and feel?
2	Voice latency budgets	How 1,400 ms serial becomes ~650 ms overlapped
3	Turn-taking and barge-in	Who speaks, and how you stop when interrupted
4	Streaming the voice pipeline	How STT, LLM and TTS overlap
5	Evaluating voice agents	What to measure, and what to put in CI

Architecture in one line

Most teams should ship a cascaded MVP — streaming STT, LLM and TTS — measure mouth-to-ear, then decide if a native speech-to-speech model is worth the lock-in. Reuse the text-agent tool layer behind the LLM stage. Own interruption and cancel paths early.

Prefer WebRTC for apps. Treat telephony as a distinct codec and latency world (μ-law 8 kHz, PSTN delay, DTMF, transfers). Colocate media and models. Generate TTS at the transport's sample rate.

Draw the cascade, assign millisecond budgets totaling ~700 ms with overlap, explain barge-in, and contrast WebRTC vs PSTN. That is the whiteboard version of this page.

Noise, accents and SKUs still break real calls. Bias the STT for your catalog, let people barge in to correct an id, and keep a text channel for high-stakes confirms. Do not ship voice-only money movement.

Common mistakes

Adding STT + LLM + TTS serially and wondering why first audio is 1.4 s.
Silence-only endpointing, so thinkers get cut off.
Not canceling in-flight TTS on barge-in.
Voice-only money movement with no confirmation that survives ASR errors.
Polishing the brand voice before the pipeline is fast enough that callers stay.

Key takeaways

Cascade maximizes control. Speech-to-speech maximizes feel when the tool surface is thin.
Aim for roughly sub-second voice-to-voice by overlapping streams.
Endpointing and barge-in are product-critical.
Prefer WebRTC for apps. Treat telephony as its own world.
Eval latency, interruptions and task success together. Handle audio privacy carefully.