Latent diffusion, guidance and conditioning

Running diffusion on full-resolution pixels is expensive. Latent diffusion first compresses each image with an autoencoder, then runs the denoising in that small latent space. The model decodes the result to pixels once at the end. Classifier-free guidance then controls how strongly the output follows the prompt.

Latent diffusion

Train an autoencoder that maps an image to a compact latent z and back. Diffusion then runs on z, which has far fewer values than the pixels. The text prompt is encoded by a language model or a CLIP text encoder, and the denoiser attends to those text embeddings at each step.

This is the design behind Stable Diffusion. The savings come from the compression, and the autoencoder's quality sets a ceiling on the final image.

Classifier-free guidance

Classifier-free guidance trains one network to make two predictions. One is conditioned on the prompt. The other is unconditioned, which is done by dropping the prompt at random during training. At sampling time, the two are combined:

ε_guided = ε_uncond + s · (ε_cond − ε_uncond)

The scale s decides how far to move from the unconditioned prediction toward the conditioned one. Here is what different values do, using made-up noise predictions where ε_uncond = 0.2 and ε_cond = 0.5:

s = 0:  0.2 + 0 · 0.3 = 0.2   (ignores the prompt)
s = 1:  0.2 + 1 · 0.3 = 0.5   (plain conditional prediction)
s = 2:  0.2 + 2 · 0.3 = 0.8   (pushed past the conditional prediction)

Values above 1 extrapolate past the conditional prediction. That is why the prompt is followed more closely.

Other kinds of conditioning

Denoisers take more than a prompt:

• Class labels for ImageNet-style tasks, passed in as embeddings.
• Image-to-image. Start the reverse process from a partly noised version of an input image, not from pure noise. A strength parameter sets how much structure is kept.
• Inpainting. Keep the known pixels, and denoise only the masked region. The mask is concatenated to the network's input.
• ControlNets and adapters. Add extra inputs such as edges, depth or pose through parallel encoders.

Each one is still the same reverse process, with a richer ε_θ(x_t, t, c).

Common mistakes

• Using a high guidance scale by default. It burns highlights and repeats artifacts. Sweep it.
• Reusing one scale across models. Each model and sampler has its own sweet spot.
• Forgetting to train the unconditional branch. Classifier-free guidance needs a model that has seen prompts dropped at random.

Check yourself

1. If s = 1, what is the guided output? The conditional prediction, since ε_uncond + 1 · (ε_cond − ε_uncond) = ε_cond.
2. Why does diversity drop as the scale rises? A higher scale pushes harder toward the prompt, so samples cluster around prompt-matching outputs.

Key takeaways

• Latent diffusion runs the denoising in a compressed space, then decodes once.
• Classifier-free guidance combines conditional and unconditional predictions with a scale s.
• Higher s follows the prompt more closely, and it costs diversity.
• Other conditioning, such as inpainting masks and ControlNets, uses the same reverse process with richer inputs.

Further learning

Guidance costs two network passes per step, so fewer steps matter a lot: fewer steps, samplers and distillation.