Cascaded vs speech-to-speech

A voice agent is an LLM application that converses with speech instead of, or in addition to, text. The hard problem is not wiring up a microphone. It is staying inside a human conversational latency budget while handling interruptions, noisy audio, telephony quirks and the same tool-use and safety issues as text agents.

Two architectures dominate. Pick them by control versus feel, not by demo polish.

Cascaded: STT → LLM → TTS

Speech-to-text produces a transcript (often partial, then final). The LLM answers in text. Text-to-speech speaks the answer. You control each stage: swap the STT vendor, use your best tool-calling model, pick a brand voice.

Pros: best tooling for tools, RAG and guardrails; inspectable text; mix-and-match vendors; usually cheaper per minute at scale for many stacks.

Cons: latency stacks three systems; ASR errors become LLM errors; prosody and emotion from the user's voice are mostly discarded unless you pass extras.

Cascaded stacks are also easier to eval: you can assert on transcripts and tool calls with the same harness as text agents, then layer audio metrics on top. That alone keeps many regulated industries on cascade even when speech-to-speech sounds nicer in a demo.

Native speech-to-speech

A realtime multimodal model takes audio in and emits audio (and often a text transcript) directly — OpenAI Realtime-style and similar offerings as of when this was written (2025–2026).

Pros: lower end-to-end latency when tuned; can preserve more paralinguistic cueing; fewer moving parts.

Cons: less mature tool ecosystems in some stacks; harder to insert deterministic text guardrails mid-pipeline; cost and lock-in; debugging is "listen to the call" more than "read the JSON."

As of 2025–2026 the gap is closing on tools, but you should still verify whether your required function-calling, RAG citation and PII redaction flows are first-class before betting a call-center migration on speech-to-speech alone.

Hybrid sketch: use speech-to-speech for greeting and clarification turns. When the user asks to change an account setting, switch the session into a cascaded tool path with explicit confirmations. Harder to build. Sometimes the only way to get both feel and safety.

Ship a cascaded MVP with streaming STT/LLM/TTS, measure mouth-to-ear, then decide if speech-to-speech is worth the lock-in. Reuse text-agent tools behind the LLM stage so you do not invent a second product stack.

Common mistakes

• Picking speech-to-speech from a demo before you have checked tool calling, citations and redaction.
• Building two tool stacks, one for voice and one for chat.
• Ignoring ASR error. A cascaded LLM will faithfully act on a wrong transcript.

Check yourself

1. You need refund tools, RAG citations and PII redaction on day one. Which architecture? Cascaded. Those controls sit on inspectable text.
2. Why do regulated teams stay on cascade when speech-to-speech sounds nicer? They can eval transcripts and tool calls with the same harness as text agents.

Key takeaways

• Cascade maximizes control. Speech-to-speech maximizes conversational latency when it fits.
• Verify tools, citations and redaction before you bet a call center on a native speech model.
• Hybrid is allowed: feel for chitchat, cascade for account actions.
• Reuse the text-agent tool layer behind the LLM stage.

Further learning

Pipecat, LiveKit Agents. Next, the budget that decides if either architecture feels present: voice latency.