voice-agents

Speech in, text, model, text, speech out is the standard architecture and it discards everything not in the words: emphasis, emotion, hesitation, overlap. End-to-end models keep it by never routing through text, and pay with a token rate roughly 185 times higher and far less training data. This lesson covers the trade, and why interleaving is the pragmatic answer.

Turning a voice AI prototype into a real product means facing hard choices. Learn the two competing architectures (the debuggable cascaded pipeline versus end-to-end speech-to-speech models), how function calling lets an agent actually do things, why cost and observability drive real decisions, the failure modes that break voice agents in production, and how to evaluate whether one actually works.

The hardest part of a voice agent is not understanding words, it is the rhythm of conversation. Learn how agents detect when you have finished speaking (endpointing), handle being interrupted mid-sentence (barge-in), fill awkward silences, and why converting speech to text throws away the emotion in your voice. This orchestration is what separates a natural agent from a robotic demo.

A voice AI agent that you can call and talk to like a person is really three systems working together: speech-to-text ears, a large language model brain, and text-to-speech voice. Learn how the STT-LLM-TTS pipeline works, why latency is the central challenge, and the streaming trick that makes a machine respond fast enough to feel like a real conversation.