How diffusion learns to denoise

Diffusion turns generation into a series of small denoising steps. Training takes a clean image, adds noise in many small steps, and teaches a network to predict the noise. Sampling reverses the process. It starts from pure noise and removes a little each step, until an image appears.

The forward process: adding noise

The forward process adds a little Gaussian noise at each step, so the image gradually turns into pure noise. You do not need to run all the steps to get any step's result. The closed form jumps straight to any step t:

x_t = √(ᾱ_t) · x₀ + √(1 − ᾱ_t) · ε        ε ~ N(0, I)

Here x₀ is the clean image, ε is random noise, and ᾱ_t is a number that falls from near 1 to near 0 as t grows. Early in the process, x_t is mostly image. Late in the process, it is mostly noise.

x_t = √0.5 · 1.0 + √0.5 · 0.2 = 0.707 + 0.141 ≈ 0.849

The reverse process: predicting noise

Train a network ε_θ(x_t, t) to look at the noisy image and the step number, then predict the noise that was added. Sampling runs in reverse. The network's noise estimate tells the sampler what to subtract, and each step removes a little noise until the image is clean.

The loss from Ho et al., 2020 is short:

L = E[ || ε − ε_θ(x_t, t) ||² ]

It is the squared error between the noise that was added and the noise the network predicts. That simplicity is why the method trains stably.

Training step, as a sketch

This pseudocode shows one training step in latent space, where the image has already been compressed. It is a sketch of the logic, not a complete program.

z0 = encode(image)                          # compressed image from an autoencoder
t = torch.randint(0, T, (batch,))           # random step for each example
eps = torch.randn_like(z0)                  # the noise to add
zt = sqrt_alpha_bar[t] * z0 + sqrt_one_minus[t] * eps
pred = denoise(zt, t, text_emb)             # U-Net or transformer
loss = F.mse_loss(pred, eps)

Each step sees a random noise level, so after training the network can denoise at every level. Sampling then runs the same network many times, from pure noise down to an image.

Common mistakes

• Expecting one-step generation. Basic diffusion makes many passes over the whole image. Fewer-step samplers are a later refinement.
• Mixing up prediction targets. Some models predict the noise, others predict the clean image or a velocity. Check which one your code uses.
• Judging quality by the loss value. The loss averages over random noise levels. Look at samples.

Check yourself

1. In the closed form, what does a small ᾱ_t mean for x_t? Little of the image remains, so x_t is close to the noise ε.
2. What does the network learn to predict in the DDPM loss? The noise ε that was added to the clean image.

Key takeaways

• The forward process adds noise in steps, and a closed form jumps to any step.
• The network predicts the added noise, given the noisy image and the step.
• Sampling starts from pure noise and removes a little each step.
• The training objective is a simple squared error on the noise.

Further learning

Conditioning on a prompt and speeding up sampling are the next two steps: latent diffusion, guidance and conditioning.