Guidance and samplers: steering diffusion and making it fast

The previous lesson built a model that turns static into plausible images. A product needs two more things: the image must match a prompt, and it must arrive in seconds, not minutes.

Both problems live in the sampling loop, and both were solved without changing the core training idea:

• Steering. The denoiser must be told what to draw, and, crucially, told how strongly to obey. The mechanism, classifier-free guidance, is arguably the single most consequential trick in applied diffusion.
• Speed. Naive sampling runs the network once per timestep, up to a thousand forward passes. The fixes stack: run in a compressed latent space (each pass gets cheaper), use smarter solvers (fewer passes needed), and finally distill (a student network jumps in one to four passes).

A useful frame for the whole lesson: the trained model defines a field of arrows from noise toward data (the score, from lesson 1). Everything here is about bending those arrows toward what the user asked for, and following them in fewer, larger, smarter steps.

Making a denoiser conditional is architecturally straightforward: feed a representation of the condition into the network alongside the noisy image and timestep, $\epsilon_\theta(x_t, t, c)$.

For text-to-image, $c$ is produced by a text encoder, a frozen language model or a contrastively trained image-text encoder, whose token embeddings enter the backbone through cross-attention: every spatial location of the image representation attends over the prompt tokens and pulls in what is relevant locally. This is how "a red cube on a blue sphere" routes red to one region and blue to another. Class labels, depth maps, sketches, or reference images enter the same way, or by simple concatenation for spatially aligned conditions.

Training barely changes: the same noise-prediction loss, now with the condition attached to each example.

The catch shows up at generation time: a conditional model trained this way treats the prompt as one input among many, and the unconditional pull toward "any plausible image" often dominates. Samples come out generic, loosely related to the text. Conditioning gets the information in; it does not by itself make the model obey. That requires guidance.

Classifier-free guidance (CFG) solves obedience with an elegant two-for-one. During training, randomly drop the condition (replace the prompt with a null token, typically ~10% of the time). The single network then knows both a conditional prediction $\epsilon_\theta(x_t, c)$ and an unconditional one $\epsilon_\theta(x_t, \varnothing)$.

At sampling time, run both and extrapolate past the conditional one:

$\tilde\epsilon = \epsilon_\theta(x_t, \varnothing) + w \cdot \big(\epsilon_\theta(x_t, c) - \epsilon_\theta(x_t, \varnothing)\big)$

The difference in parentheses is a direction in noise space: "what the prompt changes about the prediction." The guidance scale $w$ amplifies it. $w = 1$ is plain conditioning; $w = 0$ ignores the prompt; typical products run $w \approx 5$ to $8$.

The trade-off is real and visible: higher $w$ buys prompt fidelity and sharper, more saturated images, but overcooks at the extreme (burned colors, reduced diversity, distorted anatomy), because you are pushing samples off the learned data manifold. That one scalar, exposed in every image tool as "CFG scale," is the practical dial between creativity and obedience. Cost note: vanilla CFG doubles compute (two forward passes per step), one more reason the speed section matters.

The first speed lever attacks the cost of each step. Pixels are a wasteful place to diffuse: a 1024x1024 RGB image is ~3 million values, most encoding texture detail that a generative model does not need to reason about step by step.

Latent diffusion (the architecture behind the Stable Diffusion family) inserts a pretrained autoencoder: an encoder compresses the image, typically 8x per side, into a latent tensor ~48x smaller; a decoder reconstructs pixels nearly losslessly. Diffusion then happens entirely in latent space: noise latents, denoise latents, and only at the very end decode once to pixels.

The wins compound:

• Every network pass processes a far smaller tensor, so the same compute buys a bigger, smarter backbone.
• The autoencoder absorbs perceptual detail (skin texture, grain), letting the diffusion model spend capacity on composition and semantics.
• Training datasets can be pre-encoded once, shrinking storage and I/O.

The gotcha inherited by every latent system: the autoencoder is a ceiling. Whatever it cannot reconstruct, no amount of diffusion quality can restore; historically that meant small text and fine faces suffered first. Modern systems raised the latent channel count largely to lift this ceiling.

The second lever reduces how many steps you take. Recall from lesson 1 that a trained model defines a differential equation from noise to data; a "sampler" is just a numerical solver for it, and solvers differ enormously in how large a step they can take accurately.

• DDPM (ancestral) sampling: the original stochastic chain, one small noisy step per timestep. Faithful but needs hundreds of steps.
• DDIM reinterprets the same trained model as a deterministic ODE: same start point, same endpoint distribution, no injected noise along the way. Steps can be much larger; 20-50 suffice. Determinism also gives reproducibility and meaningful latent interpolation.
• Higher-order solvers (Heun-style, DPM-Solver++ and family) use multiple function evaluations or history to cancel discretization error, pushing decent quality to ~10-20 steps.

The key conceptual point: none of these require retraining. The model is fixed; only the integration scheme changes. Below ~10 steps, however, solver cleverness hits a wall, the arrows genuinely curve, and big straight steps miss. Getting to 1-4 steps needs training-time interventions.

To go below the solver wall, change the model, not the solver. The family of techniques is distillation: train a student to reproduce, in one big jump, what the teacher achieves in many small ones.

• Progressive distillation: the student learns to match two teacher steps with one of its own; halve repeatedly (1000 to 500 to 250 ... to 4). Each round is cheap supervised regression.
• Consistency models: train a network mapping any point on a noise trajectory directly to that trajectory's endpoint, with a self-consistency loss (any two points on the same trajectory must map to the same result). One step generation, a few steps to refine.
• Adversarial distillation (used by "turbo"-style checkpoints): add a discriminator so the student's single-step outputs stay sharp; crisper than pure regression students at very low step counts.

Costs, stated honestly: distilled students trade a little diversity and prompt fidelity for speed, guidance often has to be baked in (the CFG knob may be fixed or limited), and each distillation is an extra training run. Interactive products accept those trades happily, and this is how "real-time" image generation actually works.

One stone remains unturned: if big straight jumps are what we ultimately want, why train along curved paths at all? Making straightness the training objective itself is the idea behind flow matching, the next lesson.

Put the pieces together and you can read any modern text-to-image system as one pipeline:

prompt
  └─ text encoder ──────────────► token embeddings
noise latent (e.g. 128x128x C)
  └─ DiT / U-Net backbone ◄──── cross-attention on embeddings
       │  repeated N times (N = 4-50 by product tier)
       │  each step: eps_cond, eps_uncond → CFG mix (scale w)
       ▼
clean latent ── autoencoder decoder ──► 1024x1024 image

And the knobs you now understand mechanically, not just as sliders:

• Steps: which solver regime you are in (or whether the model is distilled).
• CFG scale: extrapolation strength between unconditional and conditional predictions.
• Seed: the initial noise latent; with a deterministic sampler, same seed + prompt = same image.
• Negative prompt: literally swaps what fills the unconditional slot in the CFG formula, so guidance pushes away from it. A direct corollary of the math, not a separate feature.

Every item traces back to two ideas: bend the arrows (guidance), and follow them cheaply (latents, solvers, distillation).