Somewhere around 300 milliseconds is where an AI voice agent stops feeling like a person and starts feeling like a machine you’re waiting on. Cross 600 milliseconds and callers start jabbing buttons like it’s an old phone tree. Past a second and a half, they hang up. Every architectural decision in a voice agent stack ultimately serves that one number, and most of what makes this category hard is squeezing latency out of a pipeline that has several places to lose it.
This is a practical guide to building an AI voice agent in 2026: the architecture choice that decides everything else, the latency budget you’re actually working with, real cost numbers, and the mistakes that kill projects before they ship.
In this guide:
- The Latency Budget That Decides Everything
- Cascade vs Speech-to-Speech
- The Voice Agent Stack, Layer by Layer
- Managed Platform or Open Framework?
- Real Costs
- Five Mistakes That Kill Voice Agent Projects
- The Bottom Line
The Latency Budget That Decides Everything
For any AI voice agent, sub-500 millisecond perceived latency is achievable with disciplined streaming at every stage of the pipeline, but the industry-published median for production voice agents actually sits at 1.4 to 1.7 seconds, with P99 latency running 3 to 5 seconds. That gap between what’s achievable and what’s typical is almost entirely an engineering discipline problem, not a fundamental limit.
The single biggest lever is LLM time-to-first-token — how quickly the model starts producing output after it receives the transcribed input. Everything else in the pipeline (transcription, audio synthesis) can be heavily optimized and still lose the whole latency budget if the LLM stage is slow to start streaming.
This is why voice agent architecture decisions look different from typical LLM application decisions. A model that’s excellent for chat but slow to start streaming can sink an entire voice pipeline regardless of the quality of its eventual output.
Cascade vs Speech-to-Speech
Every AI voice agent architecture is one of two approaches, and the choice shapes everything downstream.
Cascade (STT → LLM → TTS) transcribes speech to text, sends that text to a language model, and synthesizes the response back to speech — three discrete stages, three network hops in the naive version. This remains the dominant production pattern in 2026, and for a specific reason: it gives you full visibility and control at each stage, which matters enormously for tool-calling reliability and observability. You can log the exact transcript, inspect exactly what the LLM decided, and debug a bad response by looking at the text in the middle.
Speech-to-speech processes audio directly without a text intermediate stage — OpenAI’s Realtime API and Gemini Live are the reference examples. It can achieve lower raw latency by skipping stages, and it captures paralinguistic nuance (tone, emotion, hesitation) that a text intermediate step discards. What it sacrifices is exactly what cascade provides: transparency into tool-calling behavior and straightforward debugging when something goes wrong.
The pattern most production agents converge on in 2026 is hybrid: speech-to-speech for natural chit-chat and conversational filler, cascade for anything involving tool calls or actions with real consequences. Pick per-segment rather than committing the whole system to one architecture.
The Voice Agent Stack, Layer by Layer
Speech-to-text. Modern STT engines report word error rates in the mid-single digits on production call data, with strong noise robustness and broad language coverage now standard rather than a differentiator. Latency for this stage alone is typically well under 300 milliseconds when properly streamed.
The LLM. This is where time-to-first-token matters most, as covered above. The model choice here should be evaluated specifically for streaming responsiveness, not just for general chat quality — a model that’s excellent in a text chat interface can be a poor fit if it’s slow to begin streaming.
Text-to-speech. The leading streaming TTS models in 2026 report time-to-first-audio figures in the tens of milliseconds under benchmark conditions, with word error rates in the low single digits. Voice quality and language coverage vary meaningfully between providers, so this is worth auditioning directly with real conversational text rather than trusting a spec sheet.
Turn-detection and interruption handling. How an agent handles a caller talking over it — barge-in — affects perceived conversational quality more than raw latency numbers do. This needs to be tested with your actual assembled pipeline, not evaluated component-by-component, because turn-detection behavior emerges from how the pieces interact.
Managed Platform or Open Framework?
Below roughly 10,000 minutes a month, a managed platform like Vapi or Retell is the better call. You configure your STT, LLM, and TTS providers through a dashboard or API, and the platform handles the realtime orchestration loop — listening, thinking, speaking — along with telephony integration, connection management, and audio transport. Time to a working phone-number demo is measured in hours, not days.
Above that volume, building directly on an open framework like LiveKit or Pipecat typically undercuts managed platforms by 60 to 80% per call, because you’re no longer paying a platform margin on top of the underlying provider costs. The trade is engineering time — you own the orchestration, the scaling, and more of the failure-mode handling yourself.
Within the open-framework camp: LiveKit suits WebRTC-native or multi-participant scenarios and offers a mature room model that’s hard to replicate from scratch. Pipecat is a Python-first, fully open-source framework that abstracts pipeline coordination into a composable event loop, with a large plugin ecosystem across STT and TTS providers. Both let you choose any voice layer and any LLM rather than working within a fixed vendor menu, which is the whole point of moving off a managed platform in the first place.
A third option worth knowing: skip a framework entirely and use a realtime speech-to-speech API directly for the simplest possible path to a working voice pipeline, when you don’t need the tool-calling transparency that cascade and framework-level control provide.
Real Costs
A custom AI voice agent built on an open framework typically costs somewhere in the $5,000 to $80,000 range to build, spread over one to four months, depending on complexity and how many custom integrations it needs. That’s a build cost, separate from ongoing per-minute usage costs across STT, LLM, and TTS providers.
Managed platforms invert that trade: lower upfront build cost, higher ongoing per-minute cost. The crossover point most teams report sits around 10,000 minutes of monthly usage — below it, the platform’s convenience outweighs its margin; above it, the margin outweighs the convenience.
One cost trap worth flagging specifically: LLM pass-through costs on managed platforms can be a surprise if you’re not watching them, since voice conversations often run longer and involve more back-and-forth turns than a typical chat session, and each turn is a fresh LLM call.
A Worked Example: Appointment-Booking Agent
Concrete numbers help more than abstract architecture talk. Consider a simple appointment-booking voice agent handling inbound calls for a small clinic.
At under 2,000 minutes a month, this is squarely managed-platform territory. A platform like Vapi or Retell gets a working phone number live within a day, with STT, LLM, and TTS wired through the platform’s dashboard rather than three separate vendor integrations. The architecture should be cascade, not speech-to-speech — booking an appointment involves checking calendar availability and writing to a booking system, both real tool calls where you want full visibility into what the agent decided and why.
Latency budget at this scale is forgiving. Callers booking an appointment tolerate a beat of thinking time in a way a customer expecting instant conversational banter wouldn’t, so the pressure to hit the tightest possible time-to-first-token is lower than it would be for a consumer-facing chat product.
The failure mode most likely to bite this specific use case isn’t latency. It’s turn-detection around interruptions — a caller correcting themselves mid-sentence, or trying to change a time they just gave — which is exactly the kind of edge case a two-week pilot with friendly internal testers won’t surface, but real callers will within the first week of production traffic.
Five Mistakes That Kill Voice Agent Projects
Optimizing one pipeline stage in isolation. A blazing-fast TTS model doesn’t help if your LLM stage is slow to start streaming. Measure and optimize end-to-end latency, not component benchmarks in isolation.
Shipping the eval suite in month six instead of week two. Voice agent rollbacks are disproportionately traced back to quality regressions nobody caught because there was no systematic way to test conversational quality before launch. Build evaluation in early, the same discipline covered in our guide to LLM evaluation tools, adapted for turn-taking and audio-specific failure modes.
Choosing speech-to-speech for tool-heavy workflows. If your agent needs to reliably call tools, check availability, or take actions with real consequences, cascade’s transparency is worth more than speech-to-speech’s latency edge. Save speech-to-speech for the conversational, low-stakes segments.
Under-budgeting for interruption handling. Barge-in behavior is easy to overlook in a demo where nobody talks over the agent, and it’s one of the first things real callers do. Test it explicitly, not incidentally.
Picking a platform before knowing your volume. The managed-versus-framework decision hinges almost entirely on expected monthly minutes. Estimate that number honestly before committing to either path, because migrating after the fact means rebuilding orchestration you already paid to build once.
AI Voice Agents: Common Questions
Do I need a phone number to test a voice agent?
Not for the LLM and TTS layers — those can be tested with recorded or live audio through a browser. But telephony introduces its own latency and audio-quality characteristics, so final validation for any AI voice agent meant for phone calls should happen over an actual phone line, not just a browser microphone.
How much does a voice agent phone call actually cost per minute?
It varies by provider stack and volume tier, but a rough production range runs from a few cents to tens of cents per minute once STT, LLM, and TTS costs are combined. Managed platforms bundle this into their per-minute rate; self-built stacks need to sum it across providers.
Can a voice agent handle multiple languages in one call?
Some STT and TTS providers support automatic language detection and mid-call switching, but reliability varies significantly by provider and language pair. Test the specific language combination you need rather than trusting a general multilingual claim.
What happens when the voice agent doesn’t understand the caller?
A well-designed voice agent has an explicit fallback path — asking for clarification, offering to repeat, or escalating to a human — rather than guessing and proceeding. This needs to be designed deliberately; it doesn’t happen automatically from picking good components.
Is speech-to-speech always going to replace cascade?
Not obviously. Speech-to-speech is improving quickly on latency and naturalness, but cascade’s transparency for tool-calling and debugging solves a different problem that speech-to-speech doesn’t obviously fix by getting faster. Expect the hybrid pattern to persist rather than one architecture fully displacing the other.
How do I evaluate a voice agent before launch?
The same discipline as any other AI system: a set of real or realistic test calls with known-good outcomes, scored consistently, run before and after every change. Add voice-specific checks — interruption handling, transcription accuracy on your actual caller accents and background noise — on top of the standard evaluation approach.
The Bottom Line
Every decision in an AI voice agent stack should trace back to the latency budget: under 300 milliseconds feels human, and everything past 600 milliseconds costs you callers. Cascade remains the right default for anything involving tool calls, with speech-to-speech reserved for the conversational segments where its latency edge and natural affect matter most.
Start with a managed platform if you’re under 10,000 minutes a month — the speed to a working demo is worth the per-minute premium at that scale. Move to an open framework once volume justifies owning the orchestration yourself, and build your evaluation suite before either milestone, not after.
Related reading: AI agent frameworks, LLM inference servers compared, and LLM evaluation tools.