Voice AI crossed a threshold in 2026 — not because the technology became perfect, but because latency got low enough and quality got high enough that real users in real workflows now prefer talking to a well-built voice agent over filling out a form. That shift changed the conversation from "is this possible?" to "how do we build it right?"
What changed in 2026
- Native speech-to-speech models arrived. GPT-4o Audio (real-time API) and Gemini Live process audio input and generate audio output without a discrete STT or TTS step, cutting the pipeline latency by ~200–400ms and preserving tone and emotion.
- Turn detection improved dramatically. Silero VAD and the detection built into native speech APIs now handle natural pauses, filler words, and interruptions with much higher accuracy.
- Telephony integration matured. Providers like Twilio, Vapi, and LiveKit have production-ready SDKs for PSTN and VoIP voice agents — the plumbing is no longer the hard part.
- Streaming became the default. Both TTS and LLM responses stream token-by-token so audio playback starts while generation is still running, shaving perceived latency.
Two architectures: pipeline vs. native
Pipeline architecture (STT → LLM → TTS):
User audio → STT (Deepgram/Whisper) → text
text → LLM (Claude/GPT-4o) → response text
response text → TTS (ElevenLabs/Cartesia) → audio
Controllable, debuggable, works with any LLM. Typical end-to-end latency: 1.2–2.5s without tuning, 600–900ms with streaming and early audio generation.
Native speech-to-speech:
User audio → Speech model (GPT-4o Audio / Gemini Live) → Response audio
Lower latency (400–700ms), better prosody, but less control — you can't intercept and modify the text mid-stream. Harder to log and evaluate.
Latency budget breakdown (pipeline)
| Stage |
Typical latency |
Optimization |
| STT (streaming) |
100–250ms |
Use streaming endpoint, not batch |
| LLM first token |
200–400ms |
Smaller model, prompt caching |
| TTS first audio chunk |
80–150ms |
Streaming TTS, short sentences first |
| Network round trips |
50–150ms |
Collocate services, use WebSockets |
| Total (optimized) |
~600–800ms |
|
The goal is to start playing audio before the LLM finishes generating — which requires streaming at every layer.
Turn detection: the unsolved problem
Natural conversation has pauses that aren't turn-endings. "I want to book a flight to... uh... New York" contains a mid-sentence pause. A voice agent that jumps in during "uh" will infuriate users.
Approaches:
- Voice Activity Detection (VAD) with configurable silence thresholds (300–700ms). Tune per use case — longer for elderly users, shorter for quick-response bots.
- Semantic turn detection — feed the partial transcript to a fast classifier that predicts whether the user has finished a complete thought.
- User-controlled push-to-talk for high-stakes contexts (medical, legal) where interruptions are unacceptable.
How to pick the right architecture
- Latency requirements < 600ms? Native speech-to-speech (GPT-4o Realtime API or Gemini Live).
- Need to log and debug every turn? Pipeline — you can inspect and store the text at every stage.
- Need custom voice or brand voice? Pipeline with a fine-tuned TTS model (ElevenLabs, Cartesia).
- Budget-constrained at scale? Pipeline with a fast smaller LLM (Llama 3.1 8B via Groq) for ~3× lower cost.
- Telephony / phone calls? Vapi or Twilio Voice AI abstracts the hard parts; use their SDKs.
Common mistakes
No fallback utterance. When the model doesn't know what to say, it must say something — "Let me check on that" buys time; silence breaks the call.
Forgetting speaker diarization for multi-participant calls. If multiple people speak, you need to identify who said what before feeding the transcript to the LLM.
Clipping audio at 8kHz for telephony. PSTN phone calls are 8kHz PCM, which degrades STT accuracy. Compensate with a telephony-tuned STT model.
Streaming too eagerly. Starting TTS before the LLM has generated a semantically complete phrase results in awkward audio splits. Buffer to a sentence boundary first.
Skipping evaluation on real audio. Testing with clean microphone audio hides how the system performs on phone noise, accents, and crosstalk.
What to skip
- ElevenLabs for telephony latency — their premium voices sound great but add 200–300ms; use Cartesia or Google TTS for latency-critical paths.
- Open-source Whisper on CPU for real-time — too slow; use Deepgram, AssemblyAI, or Whisper on GPU via Groq/replicate.
- Complex multi-turn memory before you've nailed single-turn quality — get one-turn voice responses right before adding memory and context management.
FAQ
What does a voice agent cost to run per call?
A 5-minute pipeline call (Deepgram STT + Claude Haiku + Cartesia TTS) costs roughly ~$0.03–0.08 per minute depending on model choices and call volume discounts.
Can voice agents handle accents reliably?
Deepgram and Google STT perform well across major accents. Whisper also handles accents well. Test explicitly on the accent distribution of your user base.
How do I handle interruptions?
Implement barge-in: when VAD detects user speech during TTS playback, stop playback immediately and process the new input. Most real-time APIs support this natively.
Should I use a smaller model to hit latency targets?
Yes — Claude Haiku, Gemini Flash, and Llama 3.1 8B via Groq can all hit <200ms first-token latency, which makes sub-800ms total latency achievable in the pipeline architecture.
Where to go next