End to End: What the Cascade Throws Away

The standard voice system is a cascade: recognition turns speech into text, a language model produces a text reply, synthesis turns that into speech. The catalogue's path on voice agents covers building one, and this lesson is about what the architecture costs.

The cost is concentrated in one place. Recognition outputs text, and text is a very narrow channel compared with what arrived.

What is lost is everything the words do not carry. Which syllable was stressed, and therefore what the sentence actually meant. Whether the speaker was hesitant, irritated, amused, or reading aloud. How fast they spoke and where they paused. Whether two people were talking at once. Non-lexical sounds that carry real information: a sigh, an intake of breath, a laugh, a hesitation noise.

All of it reaches the recogniser and none of it survives into the transcript.

The language model then reasons about a stripped representation, and synthesis has to invent delivery from scratch, which is why the previous lesson ended where it did. Information discarded at step one cannot be recovered at step three.

It is worth making the loss concrete, because in ordinary use the cascade works well and the failures are specific.

Sarcasm and irony invert meaning through delivery alone. The transcript of a sarcastic remark is its literal opposite, and there is nothing in the text to signal it.

Emphasis changes reference. The same six words with the stress on different syllables can pick out different things, ask different questions, or presuppose different facts. Transcription flattens all of them into one string.

Uncertainty is usually vocal. A person who says yes with a rising, hesitant contour has communicated doubt, and the transcript says yes.

Emotional state matters when the response should differ. A distressed caller and a calm one asking the same question want different handling, and the cascade cannot tell them apart.

And overlap is discarded entirely. Two people speaking at once is a normal conversational event, and a transcript is a sequence.

None of these are edge cases in human conversation. They are how conversation works, which is why the cascade feels more limited in dialogue than in dictation.

The second problem with a cascade is structural rather than informational, and it shows up as the thing users complain about.

Three models run in sequence, and each contributes delay. Recognition must decide the speaker has finished, which requires waiting through a pause long enough to be confident it is not a mid-sentence hesitation. The language model must produce enough text to start speaking. Synthesis must generate audio.

Those add. And the endpointing decision is the awkward one, because it is a direct trade: wait longer and you interrupt people less often but respond more slowly.

Streaming helps a great deal, and the voice agents path covers how: begin transcribing before the utterance ends, begin generating before transcription completes, begin synthesising before generation completes. Done well the components overlap rather than queue.

What streaming cannot remove is the endpointing wait, because knowing whether someone has finished speaking requires evidence that only arrives with time. A human uses prosody to anticipate the end of a turn, and prosody is exactly what the cascade discarded at the first step.

So the informational loss and the latency problem have the same origin.

If routing through text is what causes both problems, the fix is to not route through text. One model consumes audio tokens and produces audio tokens, and nothing is ever transcribed.

The appeal is clear. Nothing is discarded, because the tokens carry everything. There is no cascade to accumulate latency. And the model can respond to how something was said, because that reached it.

The difficulty is the arithmetic from the first lesson. Audio at eight codebooks is roughly 600 tokens per second against about 3.2 for text, so a conversation is roughly 185 times longer as audio than as a transcript. A ten-minute exchange is a couple of thousand text tokens and several hundred thousand audio tokens.

That multiplies training cost, inference cost, and the context length required to remember what was said five minutes ago.

The second difficulty is data. The internet contains an enormous quantity of text and far less transcribed conversational speech, and a model learning language purely from audio is learning from a tiny fraction of the material a text model gets. Everything a text model knows about the world came from reading, and speech data does not contain most of it.

The design that resolves this is to use both representations rather than choosing, and it takes several forms.

The common one starts from a strong text language model and attaches audio. An audio encoder produces representations the model consumes alongside text tokens, and an audio decoder lets it emit speech. The model keeps everything it learned from text and gains an audio channel, rather than learning language from audio.

Interleaving is the training technique that makes this work. Sequences mix text tokens and audio tokens, so the model learns their correspondence and can move between them. It can reason internally in text, which is cheap and where its competence lies, while receiving and producing audio.

A reduced token rate is the other half. If a tokenizer emits fewer tokens per second, the whole problem shrinks proportionally, which is why low-frame-rate and semantically-organised tokenizers have received so much attention.

The honest summary is that end-to-end speech models today are text models with audio attached rather than models that learned language from sound. That is a compromise, and it is the one that works.

Laying the options side by side shows that each solves one problem and creates another.

The cascade routes audio through text and back. Everything paralinguistic is discarded at the first arrow, latency accumulates across three models, and in exchange every component is independently swappable, debuggable and cheap. This is what almost everything in production does.

The pure end-to-end model keeps audio throughout. Nothing is discarded and latency is one model deep, and it pays with the token rate, the data scarcity, and the loss of the text model's accumulated knowledge.

The interleaved design keeps a text backbone and adds audio at both ends. Paralinguistic information reaches the model, the text model's knowledge is retained, and reasoning can happen cheaply in text. What it costs is complexity, and the fact that the model is doing two things at once.

The diagram is worth reading as a set of trades rather than a progression. The cascade is not obsolete. For a system that transcribes dictation or answers factual questions, discarding prosody costs nothing and the cascade is simpler, cheaper and easier to fix.

One capability is genuinely out of reach for a cascade and is the clearest argument for a different architecture.

Human conversation is not strictly turn-taking. People overlap constantly: back-channel sounds while the other person is still talking, starting a reply before the speaker finishes, interrupting and being interrupted, both parties speaking and one yielding.

A cascade is structurally half duplex. It listens, decides the turn ended, thinks, then speaks. While speaking it is not really listening, and interruption handling is bolted on: detect that the user started talking, stop playback, discard the rest, go back to listening.

That works and it is a workaround. The system was never simultaneously producing and perceiving.

Full duplex means modelling both streams continuously, so the model is always receiving audio and always deciding whether to emit, including deciding to stop mid-word because the other party started talking, or to make a brief acknowledging sound without taking the turn.

That requires an architecture where listening and speaking are not separate phases, which a pipeline of three sequential components cannot provide regardless of how well it is engineered.

The metrics from the earlier lessons measure components and miss most of what makes a voice system good or bad.

Word error rate scores the recogniser. Mean opinion score and speaker similarity score the synthesiser. A system can be excellent on all three and unpleasant to talk to.

The things that decide that are conversational. Time to first audio, which is what users experience as responsiveness, measured end to end rather than per component. Interruption handling: how quickly the system stops, and whether it correctly resumes or abandons what it was saying. Turn-taking errors in both directions, meaning talking over the user and leaving awkward silences. And whether the response is appropriate to how something was said, not only to what was said.

None of these are captured by component metrics, and all of them are what a user notices in the first minute.

The general shape is the one this catalogue keeps arriving at. Component metrics are regression tests, useful for catching things that broke. Whether the system is good is a different measurement, made end to end, on the dimension the user actually experiences.

Five things, in the order they became useful.

Audio has to be discretised before it can be modelled, and neural codecs do it with residual vector quantization: several modest codebooks each refining the last, because one codebook large enough would be untrainable.

The token rate that results governs everything. Roughly 600 tokens per second at good quality, against about 3.2 for text, is why speech models make the compromises they do.

Recognition is the easy direction, because it consumes audio and produces text. Its three architectures differ on how much audio is needed before an output, which is the same as asking whether it can stream.

Synthesis became tractable when audio turned into tokens, since generating speech then became next-token prediction. That is what made zero-shot cloning from three seconds possible, and the enabling fact is that the model already knew the space of voices.

And the cascade's cost is informational as well as temporal. Text discards everything the words do not carry, which is most of what makes conversation work, and the architectures addressing it are trading that recovery against a token rate two orders of magnitude higher.