Guide

Agents and tools

Evaluating voice agents

Score latency, barge-in, ASR impact and task success together. A scripted call suite belongs in CI next to text evals.

100 of 155 · 2 min read

Voice agents fail in ways text evals miss. A perfect tool call that arrives 1.8 seconds late still loses the caller. Evaluate the axes below together, not as an afterthought on a text harness.

Concept

Before you start: reuse the same tool and eval discipline as text agents, then add audio metrics. Cascaded stacks make this easier because you can assert on transcripts.

Axes text evals miss

  • Latency: time-to-first-audio, time-between-turns (p50/p95)
  • Interruption success rate: user barge-in stops agent audio within X ms
  • Word error impact: ASR mistakes that change the task outcome
  • Task success: same as text agents (tools, correct outcomes)
  • Conversation quality: human or judge rubrics for awkwardness, talking-over, dead air
  • Telephony-specific: DTMF, transfers, voicemail detection

Build a small scripted call suite: golden audio files with known transcripts and expected tool calls. Run them in CI when you change VAD thresholds or STT vendors — the same way you run text evals on prompt changes.

The 1.8 s → ~750 ms support-line breakdown lives on voice latency budgets. Put that kind of slice in the suite so a VAD or prompt change cannot silently give the gap back.

Pick a framework (Pipecat, LiveKit Agents, or similar) for transport maturity and ops fit. Then spend engineering time on endpointing, cancel paths and model tiering. The framework will not pick your latency budget.

Privacy

Logging raw audio has privacy and compliance weight (PCI on phone payments, health data, and similar). Prefer redaction, short retention and purpose limitation. Text transcripts alone are often enough for tool-call debugging.

Warning

Prefer redacted transcripts over raw audio for everyday debugging. An untested retention policy is not a safety control.

Order of work

  1. Streaming path plus barge-in cancel (feel)
  2. Latency instrumentation mouth-to-ear (truth)
  3. Tool calling with spoken confirms (utility)
  4. Telephony codec and region tuning if you take PSTN (reach)
  5. Speech-to-speech experiment if cascade cannot hit the budget (optional)

Most teams that reverse this order polish voices on a pipeline users hang up on.

Common mistakes

  • CI that only reruns text evals after a VAD or STT vendor change.
  • Logging raw audio with no redaction or retention limit.
  • Scoring task success only, so a 2-second gap never fails the suite.

Check yourself

  1. You swap STT vendors. What must CI run? The scripted call suite: golden audio, expected transcripts and expected tool calls.
  2. Why can transcripts be enough for tool-call debugging? The tool arguments and the text the model saw are in the transcript. Raw audio is extra privacy risk.

Key takeaways

  • Eval latency, interruptions and task success together.
  • Keep a golden-audio suite in CI for VAD and STT changes.
  • Redact and limit audio retention. Transcripts often suffice.
  • Feel and measurement first. Brand voice can wait.

Further learning

Pipecat, LiveKit Agents, Twilio on cascade latency.

Voice agents

Real-time speech AI: STT→LLM→TTS pipelines vs speech-to-speech models, latency budgets and turn-taking.

Evaluating LLM apps

A fixed set of inputs plus a way to score outputs, so you can tell whether a change helped.

Turn-taking and barge-in

VAD finds speech. Endpointing decides the turn is over. Barge-in must stop TTS and cancel the old generation immediately.