Diffusion language models: generating text all at once

Every mainstream LLM generates text the same way: predict token 1, append it, predict token 2 given token 1, and so on. This autoregressive loop has deep strengths (an exact factorization of the sequence probability, and a perfect fit for the KV-cache serving stack), but one structural cost: generation is sequential by construction. A 1,000-token answer requires 1,000 dependent forward passes; no amount of parallel hardware removes the dependency chain, because token $n$ cannot be computed before token $n-1$ exists.

The previous lessons built machinery with the opposite character: diffusion and flow models generate the whole object at once, refining every pixel in parallel over a handful of passes. Applying that to text is the obvious temptation, and it promises more than speed: a model that drafts globally can naturally infill a gap given both sides, revise what it already wrote, and enforce document-level structure, all awkward for a model that can only append.

Two obstacles stand in the way, and they define this lesson: text is discrete (you cannot add Gaussian noise to token 17,432), and the autoregressive incumbent is extraordinarily strong.

Diffusion needs a forward corruption process (lesson 1). For continuous data that was Gaussian noise; for tokens, the field converged on something a BERT veteran would recognize: masking.

The forward process of a masked diffusion model (MDM) replaces tokens with a special [MASK] symbol, independently, with a probability that grows over diffusion time. At $t=0$ the sequence is clean; at $t=1$ every position is [MASK]. That satisfies the two properties the Gaussian process provided:

• Closed-form corruption at any level: to noise a training sequence to level $t$, mask each token independently with probability $t$ (under the simplest schedule). No chain simulation.
• A universal endpoint: the all-mask sequence, the discrete analog of pure static, which generation can always start from.

The reverse model is a bidirectional transformer: it sees the whole partially masked sequence (no causal mask, attention flows both ways) and predicts a distribution over the vocabulary at every masked position simultaneously. The training loss is cross-entropy on those predictions, a masked-language-modeling loss reweighted across corruption levels so it forms a principled generative objective (an evidence bound on the sequence likelihood), rather than BERT's fixed 15% heuristic.

Alternative discrete corruptions exist (uniform token randomization, embedding-space Gaussian noise), but masking won on simplicity and scaling behavior.

A masked diffusion model generates by starting from an all-mask canvas and repeatedly: predicting every masked position in parallel, keeping the predictions it is most confident about, and re-masking the rest for the next round. A few dozen rounds produce a full sequence, regardless of its length.

Count forward passes for a 1,000-token completion:

Autoregressive:   1,000 sequential passes (one per token)
                  each pass cheap (KV cache: attend once per new token)

Masked diffusion: ~30-60 refinement rounds (a chosen budget)
                  each pass expensive (full bidirectional attention,
                  every position recomputed, no KV cache)

The diffusion side wins the count by ~20x, loses per-pass cost by a large factor, and nets out meaningfully faster when the denoiser commits many tokens per round, with the gap growing in output length, since rounds stay roughly fixed while AR passes scale linearly.

This is no longer hypothetical. Commercial diffusion LLMs (Mercury from Inception Labs; Gemini Diffusion from Google DeepMind; Seed Diffusion from ByteDance) demonstrated throughputs in the 1,000+ tokens/second per user range on standard GPUs, roughly 5-10x speed-competitive frontier autoregressive models of comparable quality tiers, with coding as the flagship use case (latency-sensitive, structured output, heavy infilling).

The honest asterisks: each round's bidirectional pass over the full sequence makes long-context serving expensive; the rounds themselves are sequential and cannot be cached the AR way (adapting caching to bidirectional refinement is an active engineering literature); and committing many tokens per round risks incoherence between them, the knob explored next.

The subtle science of diffusion LMs lives in which tokens to commit each round. The model outputs, for every masked slot, an independent distribution given the current visible context. Committing two tokens in the same round means neither saw the other, so jointly unlikely combinations slip through ("the the", inconsistent variable names, contradictory clauses). Committing one token per round restores full dependency, but that is autoregression, at bidirectional prices.

So decoding strategy is a genuine dial:

• Confidence-based unmasking (the workhorse): commit the top-$k$ most confident predictions per round; uncertain, dependency-laden positions resolve later with more context.
• Adaptive schemes: vary $k$ by round, by measured confidence margins, or by attention structure, committing aggressively where the text is locally independent (boilerplate, code syntax) and cautiously where it is not.
• Remasking / self-correction: unlike AR, a committed token need not be final; schemes that re-mask low-confidence committed tokens give the model something autoregression fundamentally lacks, the ability to reconsider.
• Block / semi-autoregressive hybrids: generate a block in parallel, condition the next block on it, blending AR's coherence with diffusion's intra-block speed, the direction several production systems took.

The unifying picture: autoregression and pure parallel diffusion are the two endpoints of one axis, and practical systems tune where to sit on it, per task and even per position.

Speed gets the headlines; the structural differences may matter more over time:

• Native infilling. Fill a hole given both sides: edit the middle of a function, complete a form, patch a paragraph. Autoregressive models need special training tricks (fill-in-the-middle reorderings) to approximate what bidirectional denoising does by construction.
• Revision as a first-class operation. Re-mask any span of a draft and regenerate it in context, arbitrarily many times. Iterative refinement of a whole document is the model's native mode, not a bolted-on loop of rewrites.
• Any-order and constrained generation. Fix arbitrary tokens in advance (a rhyme scheme, a JSON schema's punctuation, required keywords) and let the model fill around them; the constraint sites are just unmasked positions.
• Bidirectional planning. Every prediction conditions on the full draft state, left and right. Long-range agreement (a conclusion consistent with an introduction) is enforced continuously rather than hoped for from a one-way pass.

A useful mental model for the whole family: autoregression is writing with a pen, committed ink, one direction; masked diffusion is sculpting from mist, the whole shape emerges at once, any part revisable. Some tasks reward the pen; drafting, editing, and structured generation reward the sculptor.

Where the two paradigms actually stand, mechanism by mechanism:

Two framing errors to avoid. First, "diffusion LMs are just faster": the per-pass economics mean they win only when parallel commitment works, and long contexts blunt the edge. Second, "AR won, case closed": the demonstrated systems established that discrete diffusion scales, and once a paradigm clears that bar, its structural advantages (revision, infilling, constraints) compound with engineering investment.

The likeliest medium-term picture is not replacement but portfolio: autoregression where maximum quality per token matters, diffusion where latency, structure, or editability dominates, and hybrids blurring the boundary from both directions.

This path opened with a single question: how do you turn noise into data? The answer, assembled across four lessons, is one recipe with interchangeable parts:

1. Choose a corruption whose endpoint is trivial to sample: Gaussian mixing for pixels (lesson 1), token masking for text (this lesson).
2. Choose the path geometry: curved and stochastic (diffusion), or straight for cheap integration (flow matching, lesson 3).
3. Train by regression to point backwards along the path: predict the noise, the score, the velocity, or the masked tokens, four faces of the same objective.
4. Spend inference compute wisely: solvers, latent spaces, guidance scales, distillation (lesson 2), or confidence-ordered unmasking (this lesson). Sampling strategy is where raw capability becomes a product.

The deeper lesson is about how this field moves: a mechanism proven in one domain (image denoising) was re-derived for another (discrete sequences) by swapping the corruption while keeping the logic. Whatever generative architecture headlines next, you can now read it the same way: find the corruption, find the path, find the regression target, and find where the inference compute goes.