Diffusion models: learning to create by learning to denoise

Every generative model must answer one question: how do you turn random noise into a sample that looks like it came from your dataset? GANs answer with an adversarial game, autoregressive models with one-token-at-a-time prediction. Diffusion models answer with a trick of striking indirection: destroying data is easy, so learn to run destruction backwards.

The recipe has two halves:

• A forward process that needs no learning at all: take a real image and add a little Gaussian noise, over and over, until nothing but static remains.
• A reverse process that is learned: a neural network trained to undo one small step of that corruption at a time.

Generate by starting from pure static and applying the learned denoiser step by step until an image crystallizes. The insight that makes this practical: while "generate an image from nothing" is a monolithic, intractable task, "remove a little noise" is a small, supervised, learnable one, and you have infinite training data for it, because you manufactured the corruption yourself.

Formally, the forward process is a fixed Markov chain over timesteps $t = 1 \dots T$ (classically $T = 1000$). Each step mixes the current image with fresh Gaussian noise:

$x_t = \sqrt{1-\beta_t}\, x_{t-1} + \sqrt{\beta_t}\, \epsilon, \qquad \epsilon \sim \mathcal{N}(0, I)$

where $\beta_t$ is a small variance from a predefined noise schedule. Two properties do all the work:

• Closed-form shortcut. Gaussians compose, so you can jump straight from the clean image $x_0$ to any noise level: $x_t = \sqrt{\bar\alpha_t}\,x_0 + \sqrt{1-\bar\alpha_t}\,\epsilon$, with $\bar\alpha_t$ a cumulative product of the schedule. Training never needs to simulate the chain step by step.
• Guaranteed endpoint. As $t \to T$, every image converges to the same isotropic Gaussian, no matter what it started as. That gives generation a universal, samplable starting point.

The schedule (linear, cosine, and friends) controls how information dies: too fast and the model must learn huge jumps; too slow and early steps waste capacity on imperceptible changes.

The reverse process could be trained with a heavyweight variational bound, and the original derivations did exactly that. The landmark simplification of DDPM (Denoising Diffusion Probabilistic Models, 2020) is that almost all of it collapses into something a first-year practitioner can read:

$\mathcal{L} = \mathbb{E}_{x_0,\, t,\, \epsilon}\left[\; \lVert \epsilon - \epsilon_\theta(x_t, t) \rVert^2 \;\right]$

In words: take a real image, pick a random timestep, noise the image with a known $\epsilon$, and train the network $\epsilon_\theta$ to predict exactly the noise you added, with plain mean-squared error.

Why this works so well compared to its rivals:

• It is a supervised regression at every step; no adversarial game, no mode collapse, no discriminator to balance. Training is famously stable.
• The random timestep sampling means one network learns denoising at every corruption level simultaneously, sharing features across them.
• Predicting $\epsilon$ (rather than the clean image directly) keeps the target's scale constant across timesteps, which conditions the optimization nicely. Modern systems often use equivalent reparameterizations (predicting $x_0$ or a "velocity" $v$), a choice of target, not of principle.

The noise-prediction objective looks like a hack; it is actually deep. A parallel line of research (score-based generative models) trains networks to estimate the score function: $\nabla_x \log p(x)$, the direction in pixel space that most increases the data's log-probability. Point an arrow from any noisy image toward "more like real data," and you can follow those arrows from static to sample.

The punchline: the two views are mathematically the same. For Gaussian corruption, the optimal noise predictor and the score are related by a simple rescaling: $\epsilon_\theta(x_t, t) \approx -\sqrt{1-\bar\alpha_t}\; \nabla_{x_t} \log p(x_t)$. Predicting the noise is estimating the score.

This equivalence pays practical dividends:

• It recasts generation as solving a differential equation (an SDE, or a deterministic ODE with the same distribution of endpoints), which is what unlocks fast samplers and, later, flow matching (both covered in the following lessons).
• It explains why a single trained model supports many different sampling procedures: they are all just different numerical solvers for the same underlying field of arrows.

Forward: fixed, learning-free corruption from image to static. Reverse: the learned denoiser applied iteratively, turning static into a new image. Training only ever sees single-step denoising tasks at random noise levels.

The entire training loop, minus the model definition, fits on a slide. This is faithful DDPM in PyTorch:

import torch

T = 1000
betas = torch.linspace(1e-4, 0.02, T)          # noise schedule
alphas_bar = torch.cumprod(1 - betas, dim=0)    # cumulative signal fraction

def train_step(model, x0, opt):                 # x0: batch of clean images
    b = x0.shape[0]
    t = torch.randint(0, T, (b,))               # random timestep per image
    eps = torch.randn_like(x0)                  # the noise we will hide
    ab = alphas_bar[t].view(b, 1, 1, 1)
    x_t = ab.sqrt() * x0 + (1 - ab).sqrt() * eps   # closed-form corruption
    pred = model(x_t, t)                        # predict the hidden noise
    loss = torch.mean((eps - pred) ** 2)        # plain MSE
    opt.zero_grad(); loss.backward(); opt.step()
    return loss.item()

No discriminator, no likelihood bookkeeping, no sampling during training. The common gotcha hides at the other end: sampling naively requires running the model once per timestep, hundreds to a thousand forward passes per image. Slow sampling is diffusion's signature weakness, and the next lesson is largely about defeating it.

What architecture is $\epsilon_\theta$? It must map an image to an equally-sized noise estimate, conditioned on the timestep.

• U-Net was the classic choice: a convolutional encoder-decoder with skip connections between matching resolutions. The skips matter enormously here, because denoising needs to preserve fine detail (low-level structure passes across the skips) while reasoning globally (through the bottleneck). Timestep $t$ enters as a sinusoidal embedding added inside every block, so one network serves all noise levels.
• Diffusion Transformers (DiT) replaced convolution with a plain vision transformer over image patches. Two reasons this took over at the frontier: transformers scale more predictably (the same scaling behavior seen in language models shows up in the diffusion loss), and they compose naturally with text conditioning via attention. Leading image and video generators are DiT-family models.

A related design decision matters just as much: whether to diffuse in pixel space at all. Most production systems do not, they compress the image with an autoencoder first and run diffusion in that smaller latent space. The next lesson covers why.

Where diffusion sits among its rivals, mechanism by mechanism:

The pattern to remember: diffusion bought training stability and sample diversity by paying with sampling compute. GANs made the opposite trade. Autoregressive models dominate text because language is naturally sequential and discrete, while diffusion dominates continuous, spatial data like images, video, and audio, though the final lesson of this path shows that boundary being tested from both sides.

Everything else in this path builds on this foundation: guidance and fast samplers (lesson 2), the flow-matching reframing that straightens the denoising path (lesson 3), and diffusion applied to language itself (lesson 4).