Flow matching: straightening the path from noise to data

The last two lessons ended at the same wall twice: a trained diffusion model traces a curved trajectory from noise to data, so taking few, large steps misses the curve, and only distillation tricks got us to one-step generation.

Flow matching attacks the root cause. Instead of defining generation as reversing a corruption process and inheriting whatever trajectories that implies, it asks directly: what is the simplest path from the noise distribution to the data distribution, and can we learn to follow it?

The object it learns is a velocity field $v_\theta(x, t)$: at every point in space and "time" $t \in [0, 1]$, an arrow saying which way a sample should move. Generation is then pure physics simulation: drop a Gaussian sample at $t=0$ and integrate the ODE $\frac{dx}{dt} = v_\theta(x, t)$ until $t=1$, where it lands on the data distribution.

If that sounds familiar, it should: lesson 1 showed diffusion sampling is also ODE integration. The difference is that flow matching gets to choose the paths, and it chooses straight lines.

Learning "the velocity field that transports one distribution to another" sounds intractable, and in general it is: the marginal field depends on the entire dataset at once.

Conditional flow matching dissolves the problem with the same move DDPM used (compare lesson 1's per-step denoising): make the target per-example. Pick one noise sample $x_0 \sim \mathcal{N}(0, I)$ and one data sample $x_1$. Declare their path to be the straight line:

$x_t = (1 - t)\,x_0 + t\,x_1$

Along this line, the velocity is constant and known exactly: $\frac{dx_t}{dt} = x_1 - x_0$. So train by regression:

$\mathcal{L} = \mathbb{E}_{x_0, x_1, t}\left[\, \lVert v_\theta(x_t, t) - (x_1 - x_0) \rVert^2 \,\right]$

The theorem that makes this legitimate: regressing on these conditional per-pair velocities yields, in expectation, the correct marginal velocity field for transporting the whole noise distribution to the whole data distribution. Where trajectories from different pairs cross, the network learns their average, which is exactly the marginal flow.

No noise schedules, no $\bar\alpha_t$ bookkeeping, no variational bound: sample two points, interpolate, regress on their difference.

The training loop makes the simplification tangible. Compare line for line with lesson 1's DDPM loop:

import torch

def train_step(model, x1, opt):            # x1: batch of real data
    b = x1.shape[0]
    x0 = torch.randn_like(x1)              # noise endpoint
    t = torch.rand(b, 1, 1, 1)             # continuous t in [0,1]
    x_t = (1 - t) * x0 + t * x1            # straight-line interpolant
    target_v = x1 - x0                     # exact velocity on that line
    pred_v = model(x_t, t.squeeze())
    loss = torch.mean((pred_v - target_v) ** 2)
    opt.zero_grad(); loss.backward(); opt.step()
    return loss.item()

@torch.no_grad()
def sample(model, shape, steps=8):         # Euler integration of the ODE
    x = torch.randn(shape)
    dt = 1.0 / steps
    for i in range(steps):
        t = torch.full((shape[0],), i * dt)
        x = x + model(x, t) * dt
    return x

Differences worth noticing: time is continuous rather than one of 1000 discrete steps; there is no schedule to tune; and the sampler is the most naive ODE solver imaginable, plain Euler, yet 8-16 steps often produce strong samples, because the paths the model was trained on were straight. Everything from lesson 2 still applies unchanged on top: CFG (learn a conditional and unconditional velocity, extrapolate between them), latent spaces, and DiT backbones.

A subtlety keeps this honest: declaring each training pair's path straight does not make the learned marginal flow straight. When random noise-data pairings cross each other (and with random pairing they cross constantly), the model averages their velocities, and averaged straight lines bend. The learned trajectories are straight-ish, much straighter than diffusion's, but not perfectly straight, which is why few-step sampling is good rather than magically one-step.

Rectified flow adds the fix: an iterative straightening procedure called reflow.

1. Train a first flow model with random pairings.
2. Generate samples with it, keeping each output paired with the exact noise that produced it. These pairs, by construction, do not cross the way random pairings do.
3. Retrain on these coupled pairs. Trajectories straighten substantially; repeat if desired.

Each reflow round trades a little fidelity for a lot of straightness, and after one or two rounds, 1-4 step generation becomes viable without a separate distillation stage. Later refinements (mean-flow-style objectives that regress the time-averaged velocity over an interval rather than the instantaneous one) push one-step quality further still. The intellectual through-line: make the geometry you want at inference time the thing you optimize at training time.

The cleanest way to see how the two frameworks relate: both train a network to point along a prescribed probability path from noise to data; they differ only in which path and which parameterization of the arrow.

Choose the Gaussian variance-preserving path in the flow-matching framework and you recover diffusion exactly: the velocity target becomes an affine function of the familiar $\epsilon$-target, one linear reparameterization apart. That is why the field talks of one unified family now: train a vector field along a chosen path; pick the path for the properties you want. Straight paths buy cheap sampling; diffusion's curved paths retain some advantages in noise robustness and likelihood evaluation.

The practical scoreboard settled quickly: leading open and commercial image generators (the Stable Diffusion 3 / Flux generation onward) train with flow-matching objectives on DiT backbones, and the pattern spread to video, audio, and molecule generation.

Frameworks win on logistics as much as elegance. The reasons flow matching became the frontier default, in rough order of weight:

• Fewer moving parts. No noise schedule to design, no discrete-time bookkeeping, no prediction-target debates ($\epsilon$ vs $x_0$ vs $v$); the objective has essentially one form. Fewer hyperparameters means cheaper large-scale experimentation, which matters most exactly at frontier scale.
• Inference economics. Straighter paths mean fewer function evaluations for the same quality, and inference, not training, dominates the lifetime cost of a deployed image model. A 4-8 step model at production quality changes serving costs qualitatively.
• Stable continuous-time training. Continuous $t$ sampling smooths the loss landscape across noise levels; practitioners report fewer schedule-related pathologies at scale.
• A cleaner theory seam. Optimal-transport language connects flow matching to a large mathematical literature; extensions (paths between any two distributions, not just Gaussian-to-data) fall out naturally, enabling image-to-image, editing, and cross-modal transport in one framework.

A fair caveat closes the account: for a given pretrained model, well-tuned diffusion with a modern solver and distillation remains competitive; head-to-head quality gaps are modest. The switch is best read as the field consolidating on the simpler, more general formulation rather than a rout on sample quality.

Zoom out and the three lessons so far tell a single story with a strikingly economical core:

1. Define a path that morphs the data distribution into something trivially samplable (Gaussian noise): curved mixing (diffusion) or straight interpolation (flow matching).
2. Train a network by regression to point along that path: predict the noise, the score, or the velocity, all linearly interchangeable.
3. Generate by integrating the learned field from noise to data, with the step count set by path geometry: hundreds for curved paths, a handful for straightened ones, one after reflow or distillation.

Every headline generator for images, video, and audio is an instance of this recipe plus the lesson-2 toolkit (latents, CFG, conditioning).

One domain conspicuously resists the recipe: text. Language is discrete, there is no obvious Gaussian to interpolate toward, and autoregressive transformers already dominate. Yet the prize is tempting: diffusion-style generation refines all positions in parallel, promising order-of-magnitude faster generation than token-by-token decoding, plus natural infilling and revision. How diffusion is being rebuilt for discrete sequences, and how far the resulting language models have come, is the final lesson.