Beyond Text: Multimodal, Safety, and Open Problems

Autoregressive multimodal models work by forcing everything into a sequence. Images get tokenized into discrete codes, interleaved with text, and predicted left to right. It works, but the framing is awkward: an image has no canonical reading order, and raster-scanning one is an arbitrary choice the model has to absorb.

Diffusion inverts the relationship. Image generation was already iterative denoising before anyone applied it to text. So a diffusion language model and an image diffusion model are running the same algorithm on different data types: start from a maximally corrupted state, refine everything in parallel over a fixed budget of rounds, stop.

That shared structure is the argument for unification. Rather than bolting a vision encoder onto a sequence model, you can treat text tokens and image content as two things a single denoising backbone refines together, with no privileged ordering for either.

The practical benefit follows the same logic as infilling did for text. A model that refines a joint state can condition on any part of it, so image-conditioned text and text-conditioned image generation stop being separate capabilities.

The extensions followed quickly once the text models scaled.

LLaDA-V adds visual instruction tuning to the masked diffusion backbone, producing a vision-language model whose language side is diffusion rather than autoregressive. It is the direct test of whether the multimodal recipe that works for autoregressive models transfers, and it largely does.

LLaDA-o pushes toward an omni model handling multiple modalities in one system, with length-adaptive generation, which matters for a reason the middle of this lesson makes concrete.

LLaDA-MoE is the orthogonal axis: sparse mixture-of-experts routing applied to a diffusion backbone, showing conditional computation composes here as it does elsewhere.

The pattern across all three is worth naming. Each takes a technique developed for autoregressive language models, visual instruction tuning, omni-modal architecture, expert routing, and demonstrates it transfers to a diffusion backbone. Individually these are incremental. Collectively they answer a strategic question: whether choosing diffusion means giving up the accumulated toolkit of modern language modelling. So far, mostly not.

Language is not the only discrete sequence worth generating, and in some domains the diffusion framing is a better fit than autoregression ever was.

Protein and biomolecular design is the clearest case. A protein is a sequence over 20 amino acids, but it has no natural generation order, and the constraints that matter are structural and global rather than sequential. Designers routinely want to fix some residues, an active site, a binding motif, and generate the rest around them. That is infilling under arbitrary constraints, which masked diffusion does natively and autoregression approximates awkwardly. Work on generative flows over discrete state spaces has been applied directly to protein co-design for exactly this reason.

Molecule and materials generation shares the structure: discrete graphs or strings with global validity constraints and no canonical ordering.

Code sits between. It is written left to right by humans but edited non-sequentially, and the dominant real-world operation is modifying a middle region while both sides stay fixed.

The unifying rule: the more your constraints are global and your ordering arbitrary, the better diffusion fits.

Here is the structural limitation that gets least attention and causes the most friction in practice.

An autoregressive model decides its own output length. It generates until it emits an end-of-sequence token, so a one-word answer and a three-page essay use the same machinery with no advance commitment.

A masked diffusion model generates into a pre-allocated window of mask tokens. You must choose the number of positions before generation starts, because those positions are what the model denoises. Length is an input, not an output.

The consequences are awkward in both directions. Allocate too few positions and the answer is truncated mid-thought. Allocate too many and the model must fill the surplus with padding, which wastes compute proportional to the over-allocation and can degrade quality as the model stretches content to fit.

The available mitigations are partial. Predict the required length with a separate model or heuristic. Train the model to emit padding cleanly and trim afterward. Or use semi-autoregressive block decoding and simply stop adding blocks, which is the most common answer and is really a concession that the autoregressive stopping rule was useful.

Every alignment technique in wide use was designed against a left-to-right generation process, and several of them quietly depend on that.

The dependency is easy to miss. Safety training shapes behaviour along the trajectory the model actually takes, and for an autoregressive model that trajectory is always a growing prefix. Runtime filters inherit the same assumption: they inspect partial outputs as prefixes, and streaming moderation can stop generation partway through.

Masked diffusion violates all of it. There is no prefix. Intermediate states are partially-masked drafts scattered across the whole sequence, which are not text a moderation classifier was ever trained on. And the generation path from empty to finished passes through states the safety training distribution never contained.

Research has documented that this gap is exploitable in practice, with published work identifying safety vulnerabilities that are emergent properties of the masked generation process itself rather than of any particular model's training. The finding is not that diffusion models are inherently unsafe, but that inheriting an aligned autoregressive recipe does not inherit its safety properties.

Alignment for this paradigm has to be built against the decoding process it actually uses.

Comparisons in this literature are unusually easy to get wrong, and three specific errors account for most of it.

Perplexity is not comparable. As established earlier, diffusion models report an upper bound on log-likelihood while autoregressive models factorize exactly. The numbers are on different scales and the gap size is unknown.

Speed without quality is meaningless, and both depend on one dial. The number of denoising rounds, often reported as NFE (number of function evaluations), trades directly against output quality. A model can be made arbitrarily fast by cutting rounds. Any speed claim without a matched-quality comparison, and any quality claim without the NFE stated, is uninterpretable.

Benchmarks assume sequential generation. Standard harnesses score a continuation given a prompt and often compute per-token likelihoods, which requires adaptation for a bidirectional model. Details of that adaptation can move scores materially.

The minimum honest reporting is therefore: NFE, block size, confidence threshold, generation window length, and hardware. The absence of any of these in a speedup claim is the signal to discount it.

The choice is not about which paradigm is better in the abstract. It follows from whether your task has a natural generation order and whether your constraints are local or global.

The 2026 survey literature converges on a consistent list of what is unresolved.

Long sequences remain expensive. Every advantage narrows as context grows, because bidirectional attention over a full window is the cost that parallelism was supposed to amortize.

Infrastructure is immature. Years of autoregressive serving optimisation, paged attention, continuous batching, cross-request prefix sharing, speculative decoding, have to be redesigned rather than reused.

Variable-length generation is unsolved. The fixed window is a genuine architectural constraint with only partial workarounds.

Alignment needs rebuilding, for the reasons above.

Evaluation lacks agreed protocol, which makes the literature harder to read than it should be.

One encouraging development is that interpretability is arriving alongside rather than after: researchers have begun applying sparse autoencoders to diffusion language models, so the mechanistic toolkit is being extended to this architecture while it is still young rather than retrofitted a decade later.

Stepping back across the path, four claims are well supported and worth separating from the enthusiasm around them.

Discrete diffusion scales. Masked diffusion trained to 8 billion parameters is competitive with equivalently-sized autoregressive models. That is the bar that kills most alternatives, and it was cleared.

The objective is familiar. The variational bound reduces to masked language modelling generalised across masking ratios and reweighted, which is why the paradigm could adopt existing infrastructure so quickly.

The speed advantage is real but conditional. It requires approximate caching plus confidence-aware decoding to materialize at all, and it narrows at long context.

Some capabilities are structural. Native infilling, revision through re-masking, and freedom from the reversal curse follow from bidirectional training rather than from scale, and no amount of autoregressive engineering produces them for free.

The honest forecast is a portfolio rather than a succession: autoregression where maximum quality per token and mature serving matter, diffusion where editing, global constraints, or latency dominate, and hybrids blurring the line from both directions.