Two years ago a "voice agent" was a demo stitched from a phone number, a transcription API, an LLM call, and a text-to-speech round trip — impressive for ninety seconds, unusable under real conditions. In 2026, voice is a production category: businesses run phone intake, scheduling, qualification, and support on agents that handle interruptions, transfer to humans, and take actions mid-call. Analysts expect conversational AI to displace tens of billions of dollars of contact-center labor this cycle.
What changed is architecture, not magic. This guide covers the three ways production voice agents are built in 2026, the latency budget that governs everything, and the failure modes that still separate shipped systems from demos.
The three architectures
Speech-to-speech: one model hears and speaks
Realtime speech-to-speech APIs (OpenAI's Realtime API is the most used; other labs ship equivalents) collapse the whole loop into a single model session: audio in, audio out, tool calls in between.
Why teams choose it: the integration surface is small, conversational quality — prosody, timing, natural interruptions — is the best in the category, and latency is excellent because nothing is glued together.
The costs: less control at every stage. You can't swap the voice for a specific brand voice from another vendor, insert deterministic business logic between "heard" and "spoke," route easy turns to a cheaper model, or run components on-prem. Per-minute costs also run higher than tuned pipelines at scale.
Modular pipelines: STT → LLM → TTS, orchestrated
The production-control path: streaming speech-to-text (Deepgram and rivals now deliver word-level streaming with sub-300ms behavior), an LLM turn with your full agent stack — tools, retrieval, guardrails — and streaming text-to-speech (modern engines begin audio in well under 200ms). Open-source frameworks — LiveKit Agents and Pipecat are the 2026 standards — handle the hard real-time plumbing: audio transport, turn detection, barge-in, and telephony bridges.
Why teams choose it: every stage is swappable and inspectable. Custom vocabulary and accents, a specific cloned voice, per-turn model routing, compliance requirements, observability — all live here. This is also where an existing text agent (your support agent, your RAG system) gets a voice without being rebuilt.
The costs: you own real-time engineering. Endpointing tuning (when has the caller finished speaking?), latency accounting across three vendors, and reconnection behavior are your problems now.
Managed platforms: rent the whole stack
Platforms like Vapi and Retell package telephony, the pipeline, and a dashboard behind an API. Genuinely the fastest path to a working phone agent — days, not weeks.
The trade: platform pricing per minute stacks on top of model costs, deep customization hits the same ceilings as any managed product, and your call data and conversation logic live in the vendor's system. The pattern I see repeatedly: validate on a platform, then move to a pipeline when volume or requirements outgrow it — which is fine, if you plan for the migration instead of being surprised by it.
The latency budget rules everything
Human conversation tolerates about a second of pause. Beyond ~1.5 seconds, callers repeat themselves or talk over the agent, and the interaction degrades fast. A useful production budget:
- Endpointing (deciding the caller finished): 100–300ms — too aggressive and you interrupt, too lazy and you feel slow;
- STT final text: effectively concurrent with speech when streaming;
- LLM time-to-first-token: 200–500ms with fast models — this is where model choice bites;
- TTS time-to-first-audio: under ~200ms streaming.
Two design consequences follow. Stream every stage — waiting for complete outputs anywhere destroys the budget. And handle slow tools honestly: a CRM lookup that takes four seconds needs a natural filler ("let me pull that up") generated before the tool call, not silence.
What still breaks in production
- Barge-in. Callers interrupt constantly. The agent must stop speaking mid-sentence, discard its queued audio, and re-listen — a solved problem in good frameworks, a disaster when hand-rolled.
- The real acoustic world. Speakerphones, cars, accents, hold music. Test with recorded production audio, not your own voice at a desk.
- Talking too much. LLMs default to paragraph answers; phone calls need one or two sentences and a check-in. Prompting for radio-style brevity is real tuning work.
- Action integrity. A voice agent that books appointments is an agent first and a voice product second: it needs the same tool permissions, confirmation steps for consequential actions, and evaluation discipline as any agent — plus evals over audio scenarios, not just transcripts.
- Escalation. The handoff to a human — with context transferred, not restarted — is a feature callers judge you on more than any AI capability.
Choosing, in practice
- Validating a use case this month: managed platform.
- Best conversational feel, modest tooling: speech-to-speech API.
- Business logic, brand voice, cost control at scale, compliance, or an existing agent stack: modular pipeline on LiveKit Agents or Pipecat.
And in all three cases, the workflow design — what the agent may do, when it confirms, when it escalates — matters more than the voice stack. That part is scoped before any code: that's the conversation to start with.