audio

Turning a continuous signal into numbers is safe only above a specific rate, and below it the damage is silent and permanent. This lesson derives the Nyquist limit, shows exactly where a too-high frequency reappears, and explains why the fix has to be analogue and has to happen before the converter.

Speech in, text, model, text, speech out is the standard architecture and it discards everything not in the words: emphasis, emotion, hesitation, overlap. End-to-end models keep it by never routing through text, and pay with a token rate roughly 185 times higher and far less training data. This lesson covers the trade, and why interleaving is the pragmatic answer.

Once audio is a sequence of tokens, generating speech becomes the same shape of problem as generating text, and the whole language modelling toolkit transfers. That reframing produced zero-shot voice cloning from about three seconds of audio. This lesson covers the codec language model approach, why so little reference suffices, the flow-matching alternative, and what the capability implies.

Speech recognition's hard problem is that audio and text have different lengths and nobody labelled which frame goes with which letter. CTC, RNN-T and attention encoder-decoders are three answers, and which one a system uses decides whether it can stream. This lesson covers all three, why Whisper's weak supervision worked, and the failure that follows from a recogniser containing a language model.

Before a model can process speech it has to be discretised, and audio resists that harder than text does. This lesson covers why raw waveforms are the wrong representation, how neural audio codecs learn a discrete one, what residual vector quantization actually does, and the arithmetic that governs every speech model: a minute of talking is around 195 text tokens or 36,000 audio tokens.