Turning Sound Into Tokens

Text arrives already discrete. Characters are symbols, a tokenizer groups them, and the sequence is short: a minute of speech transcribed is on the order of two hundred tokens.

Audio arrives as a waveform, a continuous pressure signal sampled many thousands of times a second. At 24 kHz with 16 bits per sample, one second is 24,000 numbers, or 384 kilobits.

Two problems follow, and they pull in opposite directions.

The sequence is far too long. A transformer over raw samples would need to attend across 24,000 positions per second, which is untenable for anything but very short clips.

And the samples are the wrong unit. An individual sample carries almost no meaning. What matters is structure across thousands of them: the pitch of a voice, the shape of a vowel, the timing of a consonant. Predicting the next sample is a task about waveform continuity, not about speech.

So the first job in any speech model is finding a representation that is short enough to model and meaningful enough to be worth modelling. Everything else in this path depends on how that is solved.

For decades the answer was the spectrogram, and it is still the right starting point for understanding what came after.

Take short overlapping windows of the waveform, apply a Fourier transform to each, and keep the magnitudes. The result is a two-dimensional array: time along one axis, frequency along the other, energy as the value. Warp the frequency axis to match human hearing, which resolves low frequencies finely and high ones coarsely, and you have a mel spectrogram.

This is a very good representation. It shortens the sequence by an enormous factor, since one frame covers ten or twenty milliseconds rather than one sample. It aligns with perception. And speech structure is legible in it, to the point that a trained person can read words off a spectrogram.

Its limitation is that it is continuous. Each frame is a vector of real numbers, so a model producing speech has to regress those values, and generative modelling of continuous vectors is harder than predicting from a fixed set.

It also throws away phase, so reconstructing a waveform from a spectrogram is lossy and needs a separate vocoder.

Discretisation is what removes both problems at once.

A neural audio codec learns a discrete representation of audio rather than specifying one by hand. The architecture has three parts and is trained end to end.

An encoder, a convolutional network that consumes the waveform and produces a sequence of vectors at a much lower rate. This is the downsampling step, and the output rate is a design choice with large consequences.

A quantizer, which maps each continuous vector to the nearest entry in a learned codebook, replacing it with that entry's index. This is the step that makes the representation discrete, and the indices are what a downstream model treats as tokens.

A decoder, which reconstructs the waveform from the quantized sequence.

Training minimises reconstruction error, and adds an adversarial loss from a discriminator trained to tell real audio from reconstructed. That second term matters more than it sounds: reconstruction error alone produces muffled output, because averaging is the safe way to minimise it. The discriminator penalises exactly that, since averaged audio is easy to identify as fake.

SoundStream and EnCodec are the systems that established this design, and essentially every speech model since builds on one of them.

The obvious quantizer uses a single codebook, and it fails for a reason worth understanding, because the fix is the central idea of the field.

Quality depends on how finely the codebook covers the space of possible vectors. Finer coverage needs more entries. But the number of bits per token is the logarithm of the codebook size, so getting one more bit of precision means doubling the codebook.

Reaching acceptable speech quality this way would need a codebook with an astronomically large number of entries, which is impossible to store and impossible to train, since every entry needs to be visited enough times to learn a useful value.

Residual vector quantization solves it by using several modest codebooks in sequence.

Quantize the vector with the first codebook, which gets you close. Compute the residual, the difference between the original and the chosen entry. Quantize that residual with a second codebook, which corrects part of the error. Take the new residual, quantize with a third, and so on.

Each stage refines what the previous ones left. Eight codebooks of a thousand entries give the precision of one codebook with a thousand to the eighth power entries, and are entirely trainable.

Put numbers on a standard configuration. EnCodec at 24 kHz produces frames at 75 Hz, with codebooks of 1024 entries, which is 10 bits each.

One codebook is 75 tokens per second at 0.8 kilobits per second, a 512-fold compression against the raw 384 kbps.

Four codebooks give 300 tokens per second at 3 kbps, a 128-fold compression.

Eight codebooks give 600 tokens per second at 6 kbps, a 64-fold compression, and this is roughly where quality becomes good.

Now compare against text. Speech at 150 words per minute is about 3.2 text tokens per second.

So one minute of talking is around 195 tokens as text, and around 36,000 tokens as audio at eight codebooks. The audio representation is roughly 185 times longer for the same content.

That ratio is the single most important number in speech modelling. It decides context lengths, training costs, and latency, and it is why every design in this path is in some way an attempt to reduce it.

Laying the pipeline out shows where each design decision sits and what it controls.

The waveform enters the encoder, which downsamples it to a frame rate. That rate is the first lever: lower means fewer tokens per second and less temporal detail.

Each frame vector then goes through the residual quantizer stack. The first codebook produces a coarse index and a residual; each subsequent codebook refines. The number of codebooks is the second lever: more means better reconstruction and proportionally more tokens.

The output is not one token sequence but several parallel ones, one per codebook, all at the same frame rate. That structure is awkward for a language model, which expects a single stream, and how to handle it is a real design problem covered in the synthesis lesson.

The decoder inverts the whole thing, summing the codebook entries back into a vector sequence and reconstructing a waveform.

Note what the two levers trade against each other. Frame rate controls time resolution; codebook count controls per-frame fidelity. Both multiply into the token rate, and the token rate is what everything downstream pays for.

A codec trained purely to reconstruct audio learns tokens that are good at reconstruction, which is not the same as being good to model.

The issue is what the tokens encode. Reconstruction requires everything: the words, the speaker's voice, the room acoustics, the background noise. All of it is mixed across the codebooks with no particular organisation, so a language model predicting these tokens spends capacity predicting microphone characteristics alongside content.

The useful distinction is between semantic information, meaning what was said, and acoustic information, meaning how it sounded. A model generating speech needs both, and they behave very differently: semantics carry long-range structure, acoustics are local and largely stationary within an utterance.

SpeechTokenizer addressed this by distilling semantic information into the first codebook specifically, so that the coarsest stream carries content and the later ones carry acoustic detail. That makes the first stream something close to a phonetic transcription, which is far easier to model.

The general design goal, pursued by several tokenizers since, is to make the token structure match the structure of the problem rather than only the structure of the signal.

Since the token rate governs everything downstream, reducing the frame rate is the most direct lever available, and there is active work pushing it well below 75 Hz.

The constraint is that speech has real structure at short timescales. Phonemes last on the order of 50 to 100 milliseconds, and some distinctions are shorter: the difference between certain consonants is a matter of a few tens of milliseconds in the onset. Drop the frame rate too far and those distinctions cannot be represented, so the content itself degrades rather than only the fidelity.

The trade is therefore not fidelity against cost in a simple sense. Below a certain rate you stop losing audio quality and start losing intelligibility, which is a different kind of failure.

Lower-rate tokenizers get there by making each token carry more, which usually means larger or better-organised codebooks and more capacity in the encoder.

A related design axis is streaming. A codec that needs to see a whole utterance before encoding cannot be used in a live conversation, so tokenizers built for real-time use, such as Mimi, constrain themselves to causal operations that can emit a token as soon as its frame arrives.

The representation decides what the rest of the system can be, and three consequences run through the remaining lessons.

Recognition can mostly avoid this problem. A speech recogniser consumes audio and produces text, so it needs a good encoder and never has to generate audio tokens. That is why recognition was solved earlier and more thoroughly than synthesis, and why its architectures look different.

Synthesis is where discretisation pays off. Once audio is a sequence of tokens from a fixed vocabulary, generating speech becomes the same shape of problem as generating text, and the entire language modelling toolkit applies directly. That reframing is what made zero-shot voice cloning possible.

And end-to-end speech-to-speech is where the token rate becomes the binding constraint. A model consuming and producing audio tokens over a real conversation is working with sequences roughly two orders of magnitude longer than the text equivalent, and every architectural compromise in that area traces back to this arithmetic.

With the representation established, the next lesson takes the easier direction: turning speech into text.