Turn-taking and barge-in

Voice activity detection (VAD) decides which frames are speech versus silence. Endpointing decides when the user's turn is finished so you can answer. Silence timers alone are brittle: people pause to think mid-sentence.

Get this wrong and no amount of LLM quality saves the experience. The product either interrupts constantly or sits in dead air.

Semantic endpointing

Modern stacks use semantic endpointing — models that look at the transcript so far and estimate turn completeness — not only milliseconds of silence. Too aggressive → constant interruptions. Too passive → dead air.

A pause after "I want to cancel my…" is not a turn. A pause after "Cancel order ORD-12345." more often is. The transcript, not the waveform, is what tells them apart.

Barge-in

When the user speaks over the agent, you must:

1. Detect speech quickly
2. Stop TTS playback immediately (flush buffers)
3. Cancel or ignore the remainder of the old LLM/TTS generation
4. Start listening as the new turn

Failing to cancel wastes money and causes the agent to "talk over" the user with stale audio — the classic bad demo.

t=0.0  agent speaking: "Your order will arrive on Thurs—"
t=0.2  user: "No, wait"
t=0.3  VAD fires; flush TTS; abort LLM/TTS
t=0.3  listen

If step 3 is late, the user hears "—day" after they already interrupted. Evaluate interruption success rate: user barge-in stops agent audio within X ms.

Phone agents need barge-in that works on narrowband audio and polite handling of IVR handoffs, hold music and crosstalk. Budget extra eval time.

Confirmations over a sloppy channel

Voice does not remove tool use or guardrails. It changes the UX.

• While a tool runs, speak a short wait line, play hold audio, or accept silence (bad). Start the tool as soon as intent is clear — often before TTS of the acknowledgment finishes.
• Confirm high-risk actions verbally with explicit yes/no ("I will cancel order 12345. Say yes to confirm."). Acoustic yes/no is sloppier than a button. Log the transcript and require clear affirmation.
• Redact card numbers from TTS playback even if STT heard them. Never have the agent read back full PANs.
• Do not ship voice-only for irreversible money movement without a confirmation that survives ASR errors. Keep a text channel open in the same session when stakes are high.

Prompt injection via speech is less common than via fetched pages — but voicemail and IVR trees can still contain adversarial text once transcribed. Keep the same distrust of tool results.

Common mistakes

• Silence timers only. Thinkers get cut off. Fast talkers wait.
• Not canceling in-flight TTS. The agent talks over the user.
• Treating a mumbled "yeah" as approval for a refund.

Check yourself

1. The user says "No, wait" 200 ms into the agent's sentence. What must happen by ~300 ms? VAD fires, TTS flushes, the old generation is canceled, and the mic is listening.
2. Why are silence timers a poor endpointing rule? People pause mid-sentence to think, so a timer either interrupts them or leaves dead air.

Key takeaways

• VAD finds speech. Semantic endpointing decides the turn is over.
• Barge-in must stop audio and cancel work immediately.
• Spoken yes/no is sloppy. Log it and require a clear affirmation for writes.
• Never read back full card numbers.

Further learning

Next, how the stages overlap: streaming the voice pipeline.