The traditional voice agent is three models in a row: transcribe the user's speech, generate a text reply, synthesize it. Each stage adds latency, and the sum lands somewhere users perceive as a delay. Worse, the transcript discards everything about how something was said — the hesitation, the irritation, the emphasis — so the model reasons about words stripped of their delivery.
Speech-to-speech models take audio in and produce audio out. That removes both problems at once and introduces different ones.
What changed in 2026
- Speech-to-speech reached production quality. Direct audio models became reliable enough for customer-facing deployment rather than demonstration.
- Interruption handling matured. Detecting that a user has started speaking and stopping output promptly moved from a rough edge to a solved-enough capability.
- Latency budgets got documented. Practitioner consensus formed around what response timing feels natural, which gave teams a concrete target.
- The observability gap became apparent. With no intermediate transcript, logging, moderation, and evaluation needed rethinking, and tooling started catching up.
Pipeline versus direct
|
Pipeline architecture |
Speech-to-speech |
| Latency |
Sum of three stages |
Single model round trip |
| Tone and emotion from the user |
Lost at transcription |
Preserved |
| Transcript for logging |
Free, by construction |
Requires separate transcription |
| Moderation point |
The text between stages |
Must be designed in |
| Swapping components |
Easy; each stage independent |
All or nothing |
| Cost |
Three services |
One, often more per minute |
| Debugging |
Inspect the text between stages |
Harder; audio in, audio out |
The transcript row is the tradeoff people underestimate. In a pipeline, the text between stages is where you log conversations, run moderation, apply business rules, and evaluate quality. Remove it and every one of those needs a deliberate replacement — usually a parallel transcription running alongside purely for observability, which adds back some cost.
The latency budget
Natural conversation has a turn-taking gap measured in a few hundred milliseconds. Beyond roughly a second, the pause stops reading as thinking and starts reading as a fault; users repeat themselves or talk over the system.
Everything must fit in that window: detecting the user has finished, processing, and beginning audio output. Streaming is not optional — the system must begin speaking before it has finished deciding what to say, the way people do.
End-of-turn detection is subtle and underrated. Cutting in during a natural mid-sentence pause is jarring; waiting too long for a definite silence feels sluggish. Good systems use more than silence duration, and getting this wrong is the most common reason a technically working voice agent feels unpleasant.
Interruption is the other half. When the user starts speaking, output must stop immediately, and the system must understand that whatever it had already said was heard while the rest was not. Systems that keep talking over a user fail the basic social contract of conversation, and no amount of voice quality compensates.
Common mistakes
- A pipeline for a conversational product. Chained latency is what makes it feel wrong.
- No interruption handling. The single most noticeable failure.
- End-of-turn on silence duration alone. Either cuts people off or lags.
- No transcript plan. Logging, moderation, and evaluation all need one.
- Testing on a fast local network. Real users have variable connections; test on them.
- Ignoring cost per minute. Voice pricing is time-based and adds up quickly at scale.
FAQ
Is speech-to-speech always better?
For conversation, generally yes. For workflows where you need the text anyway — transcription products, documented interactions — a pipeline may suit better.
How do I moderate audio output?
Run transcription in parallel for observability and moderation, accepting that it lags the audio slightly. Some providers offer built-in controls.
What about phone calls?
Telephony adds its own latency and audio compression. Budget for it and test over the actual carrier path, not over an internet connection.
Can I use my own voice?
Where the provider supports custom voices, and only with documented consent from the voice owner — see TTS voices compared.
Where to go next
For the components, read speech-to-text compared and TTS voices compared. For streaming mechanics generally, streaming responses in LLM apps.