Streaming the voice pipeline

The winning cascade overlaps stages. Stream STT partials. Start the LLM when partials are stable enough, or on semantic end-of-turn. As soon as the LLM emits a clause, send it to streaming TTS. Play audio as it arrives. Keep the jitter buffer small but not underrunning.

The overlap

    # Conceptual overlap — frameworks do this with frame pipelines
async for partial in stt.stream(audio):
    ui.show_partial(partial.text)

final = await stt.wait_endpoint()
async for token in llm.stream(prompt + final.text):
    buf += token
    if sentence_boundary(buf):
        async for frame in tts.stream(buf):
            player.write(frame)
        buf = ""

That snippet is a sketch, not a shipping runtime. Pipecat and LiveKit Agents give you frame-level pipelines: transport, VAD, STT, LLM, TTS and interruption plumbing. They do not pick your latency budget. The model and the streaming design do.

Fillers ("Mm-hmm", "Let me check") can mask remaining latency for tool calls — but overuse sounds fake. Prefer real speed. Use fillers when a tool truly needs hundreds of milliseconds.

Where to flush TTS

Sentence boundaries are a design choice. Flushing TTS on every comma reduces time-to-first-audio but can make prosody choppy and force awkward mid-clause cancels. Flushing on full sentences sounds better but adds latency. Many production stacks compromise on clause boundaries (periods, question marks, long commas) and tune by listening to call recordings — not by staring at charts alone.

Speculative execution is the advanced move: begin LLM work on a high-confidence partial ("Cancel order…") and abort if the final transcript diverges ("…Cancel order status"). You save hundreds of milliseconds on the common path and waste a little compute on corrections. Cap speculation to safe, read-only prep work until the final transcript lands.

Noise, accents and SKUs

Real calls include dogs, keyboards, accents and product SKUs.

• Prefer STT models with domain biasing / keyterm boosting for your SKUs and proper nouns.
• Allow easy barge-in so users can correct ("No, ORD-six…") without waiting.
• Fall back to SMS or in-app chat when confidence is low ("I heard ORD-12145—is that right?").

Own interruption and cancel paths early. Most teams that reverse the order polish voices on a pipeline users hang up on. Feel and measurement first. Timbre and brand voice can wait a sprint.

Common mistakes

• Buffering a whole utterance before any stage starts.
• Speculating a write (cancel_order) on a partial transcript.
• Flushing TTS on every token, so prosody falls apart.

Check yourself

1. A partial says "Cancel order…" and the final says "Cancel order status." What should already have been canceled? Any speculative LLM/TTS work. Do not let a write tool fire on the partial.
2. Why flush TTS on clauses rather than every comma? First audio stays early, but the voice still has some shape and cancels land on a boundary.

Key takeaways

• Stream partials → tokens → audio. Overlap the three.
• Flush TTS on clause boundaries. Tune by listening.
• Speculate only read-only prep until the final transcript lands.
• Bias STT for SKUs. Offer a text fallback when confidence is low.

Further learning

Voice agent latency optimization (ElevenLabs)Discover what voice agent latency optimiztion is, how best to optimize voice agents to remove latency, and practical methods to deploy with low-latency.ElevenLabs

Next, what to measure: evaluating voice agents.