Recognition: Three Ways to Solve the Alignment Problem

Speech recognition maps a sequence of audio frames to a sequence of characters, and the two sequences do not line up.

A second of audio is a hundred frames or so. A second of speech is perhaps three words. So the input is much longer than the output, and the ratio is not fixed: a drawn-out vowel occupies many frames, a clipped consonant occupies few.

Worse, the training data does not say which frames correspond to which output symbol. A transcript is a sentence paired with a recording, with no annotation of where each word begins.

Classical systems solved this by building the alignment explicitly, with hidden Markov models estimating frame-to-phoneme correspondences, and separate acoustic, pronunciation and language models composed together. It worked, and it required linguistic expertise, a pronunciation dictionary per language, and a great deal of engineering per component.

Everything modern is an attempt to train one model end to end without ever computing an explicit alignment. There are three ways to do that, they were invented in that order, and each has a distinct consequence for whether the system can run in real time.

Connectionist temporal classification takes the position that if you do not know the alignment, you should not commit to one.

The model emits one output per input frame, from the character vocabulary plus a special blank symbol. A predicted frame sequence is collapsed into a transcript by two rules: merge repeated characters, then delete blanks. So the frame sequence h, h, blank, e, l, l, o becomes h, e, l, l, o, and blank is what lets a genuine double letter survive.

Many frame sequences collapse to the same transcript, and that is the point. Training does not pick one. It computes the total probability of the correct transcript by summing over every alignment that produces it, efficiently with dynamic programming, and maximises that sum.

The alignment is never chosen; it is marginalised away.

CTC is simple, fast, and naturally streaming, since each frame's output depends only on the audio, so you can emit as you go.

Its limitation is the conditional independence assumption. Each output is predicted from the audio alone, with no conditioning on previously emitted characters, so the model has no notion of what a plausible word looks like. In practice a separate language model is bolted on at decoding time to fix that.

The transducer keeps CTC's streaming property and removes its independence assumption, which is why it is what most on-device recognisers use.

It has three components. An encoder over the audio, essentially the same as CTC's. A prediction network, which is a language model over the output symbols emitted so far. And a joint network combining the two into a distribution over the next symbol.

That prediction network is the addition. The model's guess at the next character now depends on both the audio and what it has already written, so it can prefer spellings that form real words without an external language model.

Decoding walks a grid. At each step the model either emits a symbol, which advances the output position, or emits blank, which advances the audio position. It can emit several symbols for one audio frame, or consume many frames before emitting anything, which handles the variable rate naturally.

Crucially it never looks ahead. Everything depends on audio already seen and symbols already emitted, so it streams.

The cost is complexity. The training objective marginalises over paths through a two-dimensional grid, which is more expensive than CTC and fiddlier to implement well.

The third approach observes that if the model can attend anywhere in the audio, alignment stops being a constraint to satisfy and becomes something learned implicitly.

An encoder processes the whole utterance into a sequence of representations. A decoder generates the transcript one token at a time, attending over those representations, exactly as a translation model attends over a source sentence. This was introduced as Listen, Attend and Spell, and it is the architecture Whisper uses.

The attention weights end up being an alignment, learned as a side effect of optimising transcription accuracy, with no monotonicity imposed and no dynamic programming.

What this buys is generality. The decoder is an ordinary autoregressive language model over text, so everything that works for text generation works here, and the output can be anything expressible as a token sequence rather than only a transcription.

What it costs is streaming. The decoder attends over the entire encoded utterance, so it needs the whole utterance first. Systems built this way process fixed-length chunks, typically tens of seconds, which introduces latency proportional to the chunk and creates boundary problems where words straddle two chunks.

The three approaches differ on a single question with large consequences: how much of the audio must the model see before it can produce an output?

CTC needs one frame. Each output depends only on the frame beneath it, so emission is immediate and the system streams with minimal latency. The price is that outputs are conditionally independent of each other.

A transducer needs one frame plus everything already emitted. It still streams, and it now conditions on its own output, which is why it is the standard choice where both latency and accuracy matter.

An attention encoder-decoder needs the whole utterance, or at least a whole chunk. It cannot stream natively, and in exchange it has no alignment constraints and an unrestricted decoder.

The diagram makes the ordering visible, and it is a straight trade of latency against modelling freedom.

That trade explains the deployment split. Recognisers running on a phone, in a call, or in a live captioning system are transducers. Recognisers running over a recorded file, where latency does not matter, are attention models, and they are more accurate.

Whisper is an attention encoder-decoder, which is an old architecture. What was new was the data strategy, and it reversed a long-standing assumption.

Speech recognition had been trained on carefully curated corpora: recordings transcribed to a standard, verified, with consistent conventions. Such data is expensive, so datasets were small, typically hundreds or a few thousand hours.

Whisper was trained on roughly 680,000 hours of multilingual audio paired with transcripts scraped from the web. Those transcripts are imperfect. Some are automatic, some are sloppy, some are subtitles that paraphrase rather than transcribe.

This is weak supervision, and the bet was that scale would beat cleanliness.

It did, and the reason is about generalisation rather than accuracy. A model trained on clean read speech learns clean read speech and degrades on accents, noise, overlapping talkers and unusual recording conditions. A model trained on a very large sample of whatever is on the internet has seen all of that, because the internet contains all of it.

The result was a system that performed well across conditions without adaptation, which mattered more in practice than a lower error rate on any single clean benchmark.

The second design decision in Whisper is easy to overlook and is arguably the more transferable one.

Rather than training separate models for separate speech tasks, the decoder's output vocabulary was extended with special control tokens, and the task is selected by which tokens are placed in the decoder's prompt.

So one model performs language identification, transcription in the source language, translation into English, timestamp prediction, and voice activity detection. Each is a different token sequence from the same weights.

The mechanism is exactly what makes text language models multi-task: put the instruction in the sequence and let the model condition on it. Applying it to speech meant the tasks shared representations, so learning to transcribe many languages improved language identification and vice versa.

This is also why Whisper's output format is a sequence of tokens including markers rather than plain text, and why post-processing it requires knowing what the control tokens mean.

The general lesson is that adding a task to a sequence model can be a vocabulary change rather than an architecture change, which is a cheap way to get multi-task behaviour.

Attention-based recognisers have a characteristic and well-documented failure: they produce fluent text that was never said, most often on silence, music, or noise.

The mechanism follows directly from the architecture. The decoder is an autoregressive language model. Its job is to produce a plausible token sequence conditioned on the audio, and when the audio contains nothing to condition on, the language model prior takes over and it generates something plausible anyway.

The output is fluent, correctly formatted, and entirely invented. Common versions are a subtitle credit line, a repeated phrase, or a sentence carried over from earlier context.

This is worse than the classical failure mode, where an unconfident system produced garbled text that was obviously wrong. Here the failure is confident and well-formed, so downstream systems accept it.

The practical defences are all about not asking the model to transcribe nothing. Run voice activity detection first and skip silent segments. Watch for repetition, which is the most common signature. Check the model's own confidence where it is exposed. And be wary of long silences at segment boundaries.

The general point is that a component containing a language model will behave like one when its evidence runs out.

Recognition is usually reported as word error rate, the count of insertions, deletions and substitutions divided by the number of reference words. It is a reasonable summary and it hides most of what matters for a real system.

Errors are not equal. Mistranscribing an article costs almost nothing. Mistranscribing a name, a number, a drug or a command changes the meaning entirely, and both count as one substitution.

The metric is blind to who spoke. In a multi-speaker recording, a transcript with every word correct and every attribution wrong scores perfectly.

It ignores everything paralinguistic. Emphasis, hesitation, tone and overlap are absent from the reference, so a recogniser that discards them is not penalised, which matters enormously for the next lessons.

And it averages across conditions, so a system with an excellent overall figure can be substantially worse for particular accents or in particular noise conditions, which is where fairness problems live.

The useful practice is the same as everywhere else in this catalogue: report the aggregate, then report the slices you care about separately, and weight errors by what they cost you rather than by counting them.