Making Diffusion LLMs Actually Fast

Diffusion language models are sold on speed: a fixed budget of parallel refinement rounds instead of one sequential pass per token. For a long time the open implementations were slower than autoregressive models of the same size.

This was not a coding failure. It follows from the paradigm, and understanding why is the whole of this lesson.

Autoregressive serving is fast because of an accounting trick most people never examine. Generating token $n$ appears to require attending over $n$ previous tokens, which would make total work quadratic. But under a causal mask, the keys and values of previous positions never change, because nothing later can influence them. So you compute each position's key and value once, store them, and each new token costs one incremental attention over cached state.

That is the KV cache, and it is the single optimisation that makes autoregressive inference economically viable. Roughly a decade of serving infrastructure, from paged memory management to prefix sharing, is built on top of it.

Diffusion language models cannot use it, at least not as written.

Two properties of masked diffusion each independently break the caching argument.

Attention is bidirectional. Every position attends to every other, so position 5's key and value depend on the content at position 500. There is no prefix whose representations are settled.

The sequence changes every round. Each refinement step commits tokens at previously masked positions. Those positions had [MASK] embeddings before and real token embeddings now, so their keys and values are genuinely different, and by bidirectionality every other position's representation shifts too.

The cost accounting is then brutal. Let $L$ be the output length and $R$ the number of refinement rounds. Autoregressive decoding runs $L$ passes, each incremental over cached state. Diffusion runs $R$ passes, each a full recomputation over the entire sequence.

With $R$ around 30 to 60 and no caching, the per-pass cost difference can easily exceed the round-count advantage. A model doing 20 times fewer passes at 40 times the cost per pass is not faster. This is precisely the regime early implementations landed in, and it explains the gap between the paradigm's promise and its measured throughput.

Fast-dLLM, from NVIDIA researchers in 2025, restored caching without retraining anything, by conceding that the cache does not have to be exact.

The observation is empirical: although keys and values technically change every round, they change very little for positions that are already decoded. A position holding a committed token has a settled representation, and later rounds perturb it only slightly.

So divide the generation window into blocks. Decode one block at a time. While working on the current block, cache the keys and values of all previously completed blocks and reuse them across the many denoising rounds the current block requires. When the block finishes, refresh the cache once with the true values.

The cache is therefore approximate within a block and exact at block boundaries, which bounds the drift. The reported result is cache reuse with negligible quality degradation, delivering roughly 3.2x to 3.6x speedup on its own.

This is a classic systems trade: accept a small, bounded numerical error to recover an optimisation the architecture was not supposed to permit.

The second bottleneck is quality, not speed, and it decides how aggressively you can parallelise.

At each round the model outputs, for every masked position $i$, a distribution $p_\theta(x_i \mid x_{\text{visible}})$. These are marginals, each conditioned only on what is currently visible. They are not a joint distribution over the masked positions.

Committing two tokens in the same round therefore samples from the product of marginals:

$p(x_i, x_j \mid x_{\text{vis}}) \;\neq\; p(x_i \mid x_{\text{vis}})\, p(x_j \mid x_{\text{vis}})$

The two tokens never see each other. Where they are genuinely dependent, this produces the characteristic failure: duplicated words, disagreeing determiners, a variable declared with one name and used with another.

Now the useful part. If the model is highly confident about each token, with $p(x_i) > 1 - \epsilon$ for each, then almost all mass sits on one combination and the product of marginals approximates the joint to within order $\epsilon$. High confidence implies the dependency does not matter, because the conditional is sharp enough that knowing the other token would not change the answer.

That gives a principled decoding rule rather than a heuristic.

The rule that follows is to stop committing a fixed number of tokens per round and start committing however many clear a confidence threshold.

def decode_block(model, x, mask_id, threshold=0.9, max_rounds=64):
    for _ in range(max_rounds):
        masked = (x == mask_id)
        if not masked.any():
            break

        probs = model(x).softmax(-1)         # cached KV for finished blocks
        conf, pred = probs.max(-1)

        commit = masked & (conf > threshold)

        # never stall: if nothing clears the bar, take the single best
        if not commit.any():
            best = (conf * masked).argmax()
            commit = torch.zeros_like(masked).index_fill_(0, best, True)

        x = torch.where(commit, pred, x)
    return x

The behaviour adapts to the text automatically. Predictable spans, boilerplate, closing brackets, the second half of a common phrase, clear the threshold in bulk and get committed together. Genuinely ambiguous positions stay masked and resolve in later rounds with more context.

The fallback line matters: without it, a passage where nothing is confident would loop forever. Committing the single best token degrades gracefully to autoregressive behaviour exactly where autoregressive behaviour is warranted.

The reported gain from this alone is roughly 2.5x to 5.8x.

The two optimisations are independent and multiply. Caching cuts the cost of each round; confidence thresholding cuts the number of rounds needed. Combined, Fast-dLLM reports up to 11x on LLaDA and up to 27.6x end to end depending on configuration.

Block decoding, which the caching scheme requires anyway, turns out to be the same knob as the confidence threshold viewed from a different angle. Together they place any diffusion decoder on a continuous spectrum.

At one extreme, one block covering the whole output and a threshold of zero gives fully parallel generation: maximum speed, maximum risk of dependency violations. At the other, block size one gives strictly sequential generation, which is autoregression paying bidirectional prices.

Everything useful sits between. Semi-autoregressive block decoding generates a block in parallel and conditions the next block on it, which recovers long-range coherence between blocks while keeping parallelism within them.

One capability survives across the whole spectrum and has no autoregressive analogue. Because a committed token can be re-masked, the decoder can reconsider. Schemes that re-mask low-confidence committed tokens let the model repair its own earlier decisions, which is structurally impossible once an autoregressive model has emitted a token.

The practical tuning rule: lower the threshold and enlarge blocks for structured, predictable text such as code; raise it for prose where agreement matters.

Two structural problems remain after caching and thresholding, and both concentrate at long context.

Prefill is expensive. A long prompt must be processed bidirectionally, and unlike autoregressive prefill it may be revisited as decoding proceeds. Work on predictive prefilling for long-context diffusion inference targets exactly this.

Memory grows with the whole window. Autoregressive serving stores the KV cache for tokens generated so far. Diffusion serving must hold state for the entire generation window from the start, including positions still masked, because they are all attended over. Peak memory is set by the maximum output length rather than the current one.

There is also an ecosystem gap that no single technique closes. Paged attention, continuous batching, prefix sharing across requests, and speculative decoding are mature, deeply optimised, and assume causal attention. Each needs a bespoke redesign here, and work such as speculative decoding for masked diffusion is early.

So the honest position: the algorithmic gap has largely been closed by caching plus confidence decoding, while the infrastructure gap has not.

Where the two paradigms stand on inference, mechanism by mechanism:

Two gotchas to carry away. First, quoted speedups are configuration-dependent. Figures like 27.6x are end-to-end results under specific block sizes, thresholds, sequence lengths, and hardware, and they shrink substantially at long context. Always ask what was held fixed.

Second, the threshold is a quality dial disguised as a speed dial. Tuning it for throughput without measuring dependency-violation artefacts produces text that benchmarks acceptably and reads badly, because the failures are local incoherences that aggregate metrics smooth over.