Synthesis: Why Three Seconds Is Enough to Clone a Voice

Text-to-speech before neural methods was a pipeline of specialised components, and knowing its shape explains what the modern approach collapsed.

Text was normalised, expanding numbers, dates and abbreviations into words. A pronunciation dictionary and letter-to-sound rules converted words to phonemes. A duration model predicted how long each phoneme should last. A pitch and energy model predicted the prosody contour. A vocoder turned those specifications into a waveform.

Each stage needed linguistic resources built per language, and errors compounded down the chain.

The characteristic result was intelligible and unmistakably synthetic. The words were right and the prosody was flat, because prosody was being predicted by a model that saw only the text, and the text does not contain it. Which syllable a speaker emphasises, where they pause for effect, whether they sound amused, are not recoverable from the characters.

Neural methods improved each stage and then, once audio became discrete tokens, replaced the whole pipeline with a single sequence model.

The reframing is the whole idea, and it is worth stating in one sentence: if audio is a sequence of tokens from a fixed vocabulary, then generating speech is next-token prediction.

VALL-E made this concrete. Take the codec tokens from the previous lesson, and train a transformer to predict them conditioned on two things: the text to be spoken, and a short recording of the target speaker, also encoded as codec tokens.

That second conditioning input is the acoustic prompt, and it is what makes the system a voice cloner rather than a text-to-speech engine with fixed voices.

Because the residual quantizer produces several parallel token streams, the architecture splits the job. An autoregressive transformer predicts the first codebook's stream, which carries the coarse content and the timing. A non-autoregressive transformer then predicts the remaining codebooks in parallel, conditioned on the first, since those carry acoustic detail that does not need sequential decisions.

That split is a direct response to the token-rate arithmetic. Generating all eight streams autoregressively would mean 600 sequential steps per second of audio; this way only 75 are sequential.

The result that surprised people is that roughly three seconds of a speaker's audio is sufficient to produce convincing speech in their voice, on arbitrary text, with no fine-tuning.

Three seconds is nothing. It contains perhaps eight words, a fraction of the phonemes in the language, and no example of most sounds the model will need to produce.

The resolution is that the model is not learning the voice from those three seconds. It learned the space of human voices during training, from tens of thousands of hours across many speakers. That training taught it how voices vary, what dimensions they vary along, and how a given voice realises every phoneme.

The three-second prompt does not teach anything. It locates a point in a space the model already has.

This is in-context learning, exactly as a text model performs a task from a couple of examples without weight updates. The acoustic prompt is a few-shot example, and generation continues in the style it establishes.

Which also explains the failure mode. If a voice sits outside the training distribution, an unusual accent or an atypical vocal quality, three seconds cannot locate it, because there is no nearby point to locate.

A detail worth dwelling on: the acoustic prompt carries more than vocal identity, and the model reproduces all of it.

Because the codec tokens encode everything present in the recording, the prompt conveys the speaker's timbre, their speaking rate and rhythm, their emotional state, and the acoustic environment. Prompt with someone sounding cheerful in a small room and the output is cheerful in a small room. Prompt with a phone-quality recording and the output sounds like a phone call.

That is useful and it is also a source of confusion, because the environment is not usually what someone wants when cloning a voice. A prompt recorded in a reverberant space produces reverberant output on every subsequent utterance, and no amount of changing the text fixes it, because the reverberation came from the prompt.

The practical guidance follows directly. Prompt audio should be clean, close-miked, emotionally neutral unless you want the emotion, and recorded in the acoustic condition you want the output to have.

And it explains why prompt selection is a real engineering lever. Two three-second clips of the same speaker can produce noticeably different output, because they differ in everything besides identity.

Autoregressive generation is not the only option, and the alternative trades differently in a way that matters for production.

Diffusion and flow-matching approaches generate the whole utterance in parallel rather than token by token. Instead of predicting the next token given previous ones, they start from noise and iteratively refine an entire sequence toward something that looks like speech, conditioned on the text and the speaker.

The advantages are concrete. Generation is parallel, so latency does not scale with utterance length the way autoregressive decoding does. And output is more stable, because the characteristic autoregressive failure, where one bad token derails everything after it, has no analogue: there is no left-to-right dependence to derail.

That stability matters. Autoregressive speech models are known to skip words, repeat phrases, or run on, all of which are one sampling error propagating forward.

The cost is that duration must be handled explicitly. An autoregressive model decides how long the utterance is by deciding when to stop. A parallel model has to know the length in advance, so it needs a separate duration predictor, which reintroduces one piece of the old pipeline.

Both families share the same endpoints and differ entirely in the middle, which is where the trade lives.

Both take text and a speaker reference. Both end at a codec decoder turning tokens back into a waveform. The reference is encoded to tokens in both cases, because that is how speaker identity enters.

The autoregressive route generates the first codebook stream one token at a time, then fills the remaining codebooks in parallel. Variability comes free, since sampling produces a different plausible rendition each time, and that is why these models sound expressive. The exposure to derailment comes free too.

The parallel route predicts a duration, then refines the whole token grid from noise over a fixed number of steps. Latency is independent of length, output is stable, and the same input tends to produce a similar rendition, which is less expressive and more predictable.

The choice is genuinely application-dependent. An audiobook wants expressiveness and can tolerate a retry. A live agent wants bounded latency and no chance of a run-on, which is why production voice systems have been drifting toward the parallel family.

Synthesis is harder to evaluate than recognition, because there is no single correct output. Many renditions of a sentence are all valid, so there is nothing to compare against exactly.

The traditional instrument is a mean opinion score: play samples to listeners, ask them to rate naturalness from one to five, average. It measures the right thing and it is slow, expensive, and not comparable across studies, since scores depend heavily on the listener pool, the instructions and what else was in the batch.

Several automatic proxies fill in. Word error rate from running a recogniser over the output catches intelligibility failures and skipped words, and says nothing about naturalness. Speaker similarity, measured with a speaker verification model, checks whether cloning worked. Prosody and duration statistics can be compared against real speech distributions.

None of them captures the thing that actually distinguishes good synthesis, which is whether the emphasis and phrasing suit the meaning of the sentence.

So the practical position is to use automatic metrics as regression tests, catching the failures they can see, and to keep a small human listening panel for the judgement they cannot make. That is the same structure the catalogue recommends for evaluating quantization, and for the same reason.

A technique that clones a voice from three seconds of audio has consequences that are worth stating plainly rather than leaving to the reader.

The input requirement is effectively nil. Three seconds of anyone who has ever been recorded is available for a very large number of people: a voicemail, a video, a conference call, a social post.

Voice as an authentication factor is therefore substantially weakened. Systems that verified identity by voice, including some telephone banking, were relying on a property that no longer holds. The same applies to the informal version, which is recognising a family member or a colleague on a call.

Consent is structurally difficult. The reference audio can be obtained without the speaker's involvement, so unlike a recording session there is no natural moment at which permission is sought.

The mitigations that exist are partial. Detection models chase generators and lag them. Watermarking works when the generator cooperates and does not when it does not. Provenance standards for recorded media help where they are adopted.

The durable point is not a prediction. It is that a capability this cheap changes what a voice can be treated as evidence of, and systems built on the old assumption need revisiting.

Modern synthesis is very good and has a specific remaining weakness, which sets up the final lesson.

It does not know what the sentence means. The model reads text and produces a plausible rendition, and where the text is ambiguous about emphasis, it guesses. A sentence whose meaning turns on which word is stressed will be rendered with whatever stress pattern was most common in training, which may be the wrong reading.

It has no conversational context. Given a line of dialogue, it does not know what was said before, whether the speaker is responding to good news, or whether the exchange has become tense. Those determine delivery in real speech and are absent from the input.

And it cannot decide to be interrupted, to trail off, to overlap, or to make the non-lexical sounds that fill actual conversation.

All three have the same root: the input is text, and text has already discarded everything the delivery depends on. The synthesis model is being asked to reconstruct information that was thrown away upstream.

Which is exactly the argument for not going through text at all, and that is the subject of the last lesson.