Building Voice Agents: Latency, Tools, and What Breaks

The first two lessons built the concept of a voice agent: the STT-LLM-TTS pipeline and the orchestration that makes it feel natural. This final lesson is about turning that concept into something that actually works in production, handling real calls, doing real tasks, at acceptable cost and reliability. That transition forces a series of concrete engineering choices, and understanding them is what separates knowing how voice agents work from being able to reason about building one.

Four questions dominate the move from prototype to product:

• Which architecture? The modular pipeline you have seen, or a newer end-to-end approach, each with sharp trade-offs.
• How does it do things? A useful agent must not just talk but act, book, look up, update, which means connecting the LLM to real tools.
• What breaks? Production voice agents fail in specific, recurring ways that you must design against.
• How do you know it works? Evaluating a voice agent is harder than testing ordinary software, and skipping it is how bad agents reach customers.

These are the practical realities behind every deployed voice agent, from customer-service lines to phone assistants. The lesson works through each, and by the end you will understand not just the anatomy of a voice agent but the real decisions and failure modes that determine whether one succeeds. This is where the elegant architecture meets the messy world.

There is a fundamental fork in how a voice agent can be built, and it is the most important architectural decision in the field today.

The first is the cascaded (or chained) architecture from lesson one: three separate components, STT then LLM then TTS, connected through text at each handoff. Audio becomes text, text is reasoned over, text becomes audio. It is modular and, crucially, everything passes through readable text.

The second is newer: a speech-to-speech (S2S) model, a single end-to-end model that takes audio in and produces audio out directly, with no text step in between. Instead of three chained systems, one multimodal model hears and speaks, keeping everything "inside" the model as speech.

The difference is profound. The cascaded approach exposes text at each stage; the speech-to-speech approach keeps everything as speech in one model. This single distinction drives every trade-off in the next step, and understanding it is the key to the whole build decision.

The speech-to-speech approach has a compelling theoretical advantage that ties back to the last lesson: because it never converts to text, it does not throw away the paralinguistic information, the tone and emotion. In principle it can hear that you are frustrated and respond with matching warmth, and it can be faster because there are fewer stages. It sounds like the obvious future.

Yet as of 2026, the cascaded architecture dominates production deployments, while speech-to-speech is largely at the research and prototype stage. Why the mature, seemingly older approach wins in practice is not obvious, and the answer, in the next step, is a lesson in real-world engineering trade-offs that applies far beyond voice AI.

Why does the older cascaded architecture still win in production, despite speech-to-speech being theoretically superior? Because production cares about things demos do not, and on those, the text-based pipeline has decisive advantages.

The two decisive factors are observability and cost. With a cascaded agent, when a call goes wrong you get a transcript at each stage, so you can see exactly where it failed: did STT mishear the words, did the LLM reason badly, or did TTS mispronounce? A speech-to-speech model is a black box with no intermediate transcript, so when it misbehaves you often cannot tell why, which is unacceptable for a production system you must maintain and debug.

On cost, speech-to-speech models have run roughly ten times as expensive as cascaded pipelines in production, partly because the model re-processes accumulating audio context on every turn. For a service handling thousands of calls, a tenfold cost difference is decisive.

The cascaded pipeline also offers control: you can swap in a better STT, a cheaper LLM, or a new voice independently, and tune each piece. The single S2S model is all-or-nothing.

The broad lesson is one of the most important in engineering: the theoretically better technology is not automatically the practically better one. Production reliability rewards debuggability, predictable cost, and control, and the mature, modular architecture delivers those today. Speech-to-speech may well win eventually as it matures and its costs fall, and its emotional richness is a real advantage, but the current default for anyone building a real voice agent is cascaded, precisely because you can see inside it, afford it, and fix it.

A voice agent that can only chat is a novelty. A useful one does things: books the appointment, checks the order status, updates the account, transfers the call. This capability comes from function calling (also called tool use), the same mechanism that powers text-based AI agents, applied to voice.

The idea: the LLM is given a set of tools it can invoke, functions that connect to real systems. "Check availability" queries a calendar; "look up order" hits a database; "book appointment" writes to a scheduling system. When the conversation calls for it, the LLM does not just generate words, it decides to call a function, the system executes it against the real backend, and the result flows back into the conversation.

A concrete flow: the caller says "Is my order shipped yet?" The STT transcribes it, the LLM recognizes it needs data and calls the lookup_order tool with the caller's ID, the system queries the order database, the result ("shipped, arriving Tuesday") returns to the LLM, and the LLM phrases a natural spoken reply that TTS voices. The agent has now done real work, not just talked.

This is what turns a voice agent from a talking FAQ into something that can actually accomplish tasks, and it is where a voice agent connects to the rest of a business's systems.

But function calling reintroduces the enemy from lesson one: latency. A tool call takes time, querying a database or an external API can add hundreds of milliseconds or more, and it happens mid-conversation. This is exactly where the last lesson's silence-filling matters: while the lookup_order call runs, the agent should say "Let me check that for you" so the pause is covered. Function calling and orchestration are tightly linked, because the moment an agent does real work, it must gracefully manage the delay that work introduces. A capable agent is one that can both act on the world and keep the conversation feeling smooth while it does.

Between the caller and the AI sits real-time infrastructure that rarely gets attention but is essential to making a voice agent work at all. Two pieces matter most.

The first is telephony and transport: how the audio actually gets between the human and the agent. If the agent answers phone calls, it connects to the telephone network. If it runs in an app or browser, it uses real-time web protocols like WebRTC or WebSockets to stream audio with low delay. The defining requirement is moving audio bidirectionally, continuously, and with minimal lag, because, as lesson one showed, every millisecond of transport delay adds directly to the response time the caller feels. Ordinary request-response web patterns are too slow; voice needs a persistent, streaming connection in both directions at once.

The second is the reality of network conditions. Real calls happen over imperfect connections: packets arrive late or out of order, audio drops, quality varies. The infrastructure must cope gracefully, because a voice agent cannot pause and ask the caller to reload the page.

All of this connects to the latency budget from lesson one. The total delay a caller experiences is not just STT plus LLM plus TTS; it also includes the time for audio to travel over the network in both directions. As of 2026, cascaded voice agents in production typically deliver a total of roughly one and a half to three seconds from the end of the user's speech to the start of the agent's audio, depending on the models and infrastructure, and independent benchmarks of end-to-end voice systems have measured times-to-first-audio ranging from under one second to around three seconds across different systems. Shaving that delay is a constant engineering effort, and the transport layer is part of the budget.

The takeaway is that a voice agent is not just AI models; it is AI models embedded in a real-time communications system. The intelligence and the plumbing are equally necessary, and a brilliant agent on flaky, high-latency infrastructure will still feel broken to the person on the line.

Voice agents fail in specific, recurring ways, and knowing them is how you design against them. Most production problems fall into a handful of categories.

• Cascading transcription errors. If STT mishears a word, the LLM reasons on wrong input and confidently gives a wrong answer, never knowing the error came from mishearing. A single misheard name or number can derail an entire call, and the mistake originates before the intelligence even begins.
• Latency spikes. A slow LLM response, a slow tool call, or network congestion can blow past the natural-conversation threshold, producing awkward pauses that make the agent feel broken even if it is otherwise working.
• Interruption and turn-taking bugs. Endpointing that cuts people off or lags, or barge-in that fails to stop the agent, are among the most common and most infuriating real-world failures, straight from the last lesson.
• Tool failures. When a backend system is down or returns an error, the agent must handle it gracefully rather than freezing or hallucinating a made-up answer.
• Hallucination. The LLM may confidently state something false, which is worse in voice than in text, because there is no screen to scan or link to check; the caller hears a wrong answer stated with total confidence and may simply believe it.
• Emotional blindness. As covered last lesson, a cascaded agent cannot hear a caller's frustration, so it may cheerfully persist while the person grows angry, escalating a bad situation.

The pattern across these is that voice raises the stakes of every AI failure. In a text chatbot a user can re-read, scroll back, and click a source. In a live voice call, everything is transient and immediate: the caller cannot rewind, cannot see, and reacts in real time. An error that is a minor annoyance in text can end a call in voice.

Designing a production voice agent is therefore largely about anticipating and handling these failure modes, confirming critical details heard from STT, covering tool delays and failures gracefully, constraining the LLM to reduce hallucination, and building fallbacks (including handing off to a human) for when things go wrong. Robustness, not raw capability, is what makes a voice agent trustworthy.

The final challenge is knowing whether a voice agent actually works, and this is harder than testing ordinary software, which is why so many agents that seem fine in a demo fail with real callers.

Why is it hard? A voice conversation has no single right answer to check. The same request can be handled well in many different phrasings, and success depends not just on what the agent said but on the whole interaction: was it accurate, was it fast enough, did it handle interruptions, did it complete the task, did it sound natural, did it recover from errors? You cannot capture that in a simple pass-fail test.

So voice agents are evaluated along several dimensions together:

• Task success: did it actually accomplish what the caller wanted, book the appointment, resolve the issue?
• Accuracy: was the information correct, and did STT capture the caller correctly?
• Latency: were responses fast enough to feel natural, measured as real numbers?
• Conversation quality: did turn-taking, interruptions, and pacing feel human?
• Robustness: how did it handle noise, accents, confusion, and errors?

Serious evaluation uses many realistic test conversations, including messy and adversarial ones, not just the clean happy path, precisely because the happy path is exactly what a demo already covers and production is everything else. The gap between demo and production, the recurring theme of this cursus, is measured here.

Now assemble the whole picture. A voice agent is a streaming STT-LLM-TTS pipeline (lesson one) wrapped in conversation orchestration that manages turn-taking, interruptions, and silence (lesson two), connected to real tools and real-time infrastructure, built on the cascaded architecture for its debuggability and predictable cost, hardened against a known set of failure modes, and validated by multi-dimensional evaluation. The intelligence of the LLM is only one part; the engineering that surrounds it, latency, orchestration, tools, robustness, and evaluation, is what actually determines whether a voice agent feels like a helpful person or a frustrating machine. That surrounding engineering, more than the raw model, is the real substance of building voice AI.

A production agent combines the streaming pipeline, orchestration, tools, and real-time infrastructure, usually on the cascaded architecture for debuggability and cost, then hardens against failure modes and validates through multi-dimensional evaluation.