How Voice AI Agents Work: STT, LLMs, and TTS

For decades, talking to computers meant rigid voice commands: say an exact phrase, get a canned response, stray from the script and it breaks. A voice AI agent aims at something far more ambitious: a system you can call and converse with like a person, that understands natural speech, reasons about what you said, and replies in a natural voice, holding a real back-and-forth conversation.

Think of what a good voice agent does on a phone call: it answers, listens to you explain a messy problem in your own words, asks clarifying questions, looks things up, and responds naturally, ideally without you being sure whether it is human. That is a qualitatively different goal from "press 1 for billing."

Here is the key insight that organizes this whole cursus: a voice agent is not one piece of AI but several systems chained together, each doing a distinct job, plus the difficult art of making them work together in real time. Under the hood, a machine that talks is really three capabilities: hearing (converting your speech to text), thinking (deciding what to say), and speaking (converting its reply back to audio).

This lesson builds that pipeline piece by piece, then confronts the challenge that dominates everything in voice AI: latency, the delay between you finishing speaking and the agent starting to reply. Getting that delay low enough to feel natural is the central engineering problem, and the trick that solves it is the payoff of this lesson.

The standard architecture of a voice agent is a pipeline of three components, each transforming the conversation from one form to another. This chained design is the foundation of nearly every voice agent in production.

The flow, in order:

• Speech-to-text (STT), also called automatic speech recognition: takes the audio of you speaking and converts it into written text. This is the agent's ears.
• The large language model (LLM): takes that text, understands it in context, and decides what to say back, generating a text response. This is the agent's brain.
• Text-to-speech (TTS): takes the LLM's text response and converts it into natural-sounding spoken audio. This is the agent's voice.

So a full turn of conversation runs: your voice becomes text (STT), the text is understood and answered (LLM), and the answer becomes voice (TTS). Audio in, audio out, with text in the middle as the common language the components share.

The elegance of this design is modularity. Each component is a specialized system that can be built and improved independently, and swapped for a better one without rebuilding the rest. You can upgrade the STT for better accuracy, change the LLM for smarter reasoning, or switch the TTS for a more natural voice, while the overall architecture stays the same.

The rest of this lesson looks at each component's job and its challenges, then the crucial problem that emerges only when you try to run all three fast enough for a real conversation. Understanding the three parts individually is the setup; making them cooperate in real time is where voice AI gets hard.

The pipeline begins with speech-to-text (STT): converting the raw audio of human speech into written words. It is the agent's ears, and it is harder than it sounds.

Modern STT uses neural networks trained on vast amounts of speech to map sound waves to text. But human speech is messy in ways that make this genuinely difficult:

• Accents and dialects: the same words sound very different across speakers.
• Background noise: real calls happen in cars, cafes, and windy streets, not silent studios.
• Homophones and context: "their," "there," and "they're" sound identical; only meaning distinguishes them.
• Disfluencies: people say "um," restart sentences, and trail off, which the STT must handle gracefully.

A crucial feature for voice agents is that good STT works in a streaming mode: rather than waiting for you to finish your entire sentence and then transcribing it, it emits partial transcripts continuously, updating its best guess every fraction of a second (on the order of every 50 milliseconds) as you speak. So the text is taking shape in near real time while you are still talking.

This streaming behavior matters enormously, and not just for STT itself. Because the transcript is available incrementally rather than all at once at the end, the next component can start working before you have even finished speaking, a fact that becomes the key to solving the latency problem later in this lesson. For now, hold onto it: STT does not deliver one block of text at the end; it delivers a growing stream of text as you talk.

Once your speech is text, the large language model takes over as the agent's brain: understanding what you said and deciding what to say back. This is the same kind of LLM that powers text chatbots, but using it for voice introduces distinctive demands.

The LLM's job is to interpret the transcribed text in the context of the conversation so far, along with any instructions and knowledge it has been given, and generate an appropriate response. This is where the actual intelligence lives: understanding intent, remembering earlier parts of the call, reasoning about the request, and formulating a reply.

But a voice conversation is not a text chat, and the LLM must be steered to account for the differences:

• Responses must be concise. No one wants to listen to a three-paragraph spoken answer. Voice replies need to be short and to the point, closer to how a person actually talks.
• No visual formatting. Bullet lists, tables, and bold text mean nothing when spoken aloud, so the output must be plain, speakable prose.
• It must sound natural spoken. Phrasing that reads fine on screen can sound stiff or robotic when read aloud, so responses are shaped for the ear, not the eye.
• Speed matters more than ever. In text you tolerate a pause; in a live conversation, a slow reply is painfully obvious.

Just as STT streams its output, the LLM streams its response token by token as it generates, rather than producing the whole reply and only then handing it over. That means the first words of the answer are available almost immediately, before the full response exists, another piece of the latency puzzle. The brain does not think silently and then deliver a finished speech; it starts producing words as soon as it has them.

The final component, text-to-speech (TTS), converts the LLM's text reply into spoken audio, the agent's voice. This is what the caller actually hears, so its quality shapes the entire impression of the agent.

Modern neural TTS has advanced dramatically. The goal is no longer merely intelligible speech but natural-sounding speech: correct rhythm, stress, and intonation, collectively called prosody, that makes the difference between a flat robotic monotone and a voice that sounds human, with appropriate emphasis and natural pacing. A TTS that gets prosody right can be nearly indistinguishable from a recorded person.

TTS faces its own subtle challenges: pronouncing names, numbers, dates, and abbreviations correctly (does "Dr." mean "doctor" or "drive"?), and placing emphasis naturally so the meaning comes through.

And once again, streaming is central. Rather than waiting for the LLM's complete response and then synthesizing it all at once, good TTS begins speaking as soon as the first sentence (or even phrase) is ready, synthesizing and playing audio while the LLM is still generating the rest. The agent can literally start talking before it has finished deciding everything it will say, exactly as a person begins a sentence before having planned its end.

With all three components in place, ears, brain, and voice, you have a complete voice agent in principle. But running them one after another, waiting for each to fully finish before the next begins, produces something too slow to feel like a conversation. That problem, and its solution, is what the rest of the lesson is about.

Now the central problem. In a real conversation, humans expect a response almost instantly, typically within a fraction of a second of finishing their sentence. Even a one-second gap feels awkward; two or three seconds feels broken, like a bad connection or a confused listener. So the single hardest requirement for a voice agent is low latency: the delay from when you stop speaking to when the agent starts replying.

Here is why this is so hard. Look at the naive way to run the pipeline: wait for the user to fully finish, then run STT on the whole utterance, then wait for the LLM to generate its entire response, then run TTS on that full response, then play it. Each step waits for the previous one to completely finish. The total delay is the sum of every stage:

delay = detect end of speech + STT + LLM + TTS

Adding those up, a standard STT-LLM-TTS pipeline built this way commonly produces total latencies of roughly 800 to 1,500 milliseconds or more, often landing at the high end in real conditions. That is a very noticeable pause after every single thing the caller says, and it makes the conversation feel sluggish and unnatural no matter how good the individual components are.

This is the tyranny of the sequential pipeline: even if each component is individually fast, running them strictly one after another stacks their delays, and the caller waits through the whole stack after every utterance. Latency is not a minor polish item; it is the difference between an agent that feels like a conversation and one that feels like a walkie-talkie with a lag. Solving it requires abandoning the naive sequential approach, which is exactly what the next step does.

The solution to latency is the single most important idea in voice AI engineering: streaming and overlap. Instead of running the components strictly one after another, you let them work simultaneously, each starting on partial results from the one before rather than waiting for it to finish.

Recall that every component streams: STT emits partial transcripts as you speak, the LLM produces tokens as it thinks, and TTS can synthesize from the first sentence. Streaming stitches these together into an overlapping flow:

• STT is transcribing your words while you are still speaking.
• The moment you finish, the LLM already has most of the text and can begin generating almost immediately.
• As the LLM streams out its first sentence, TTS starts speaking it while the LLM is still generating the rest.

The components run in parallel, each processing the leading edge of the conversation rather than waiting for a completed handoff. The effect on latency is dramatic. Instead of the total being the sum of all stages, it collapses toward the longest single stage, roughly:

delay is closer to max(stages), not sum(stages)

This is the difference between an unnatural pause and a natural conversation. It is why the streaming pipeline is the production standard for voice agents: it is the only way a chained STT-LLM-TTS system can respond fast enough to feel human.

The deeper lesson generalizes beyond voice: when a task is a chain of stages, you slash total latency by overlapping them, streaming partial results forward, rather than running them in strict sequence. A voice agent is not just three models bolted together; it is three models running at once, each feeding the next in real time, and that orchestration is what turns a slow pipeline into a fluid conversation. The pieces are the easy part; making them flow together is the real engineering, and it sets up the even subtler challenge of the next lesson, managing the conversation itself.

Audio flows through speech-to-text, an LLM, and text-to-speech; instead of each stage finishing before the next begins, they stream partial results and overlap, so total latency approaches the slowest single stage rather than the sum.