Transformers and language models
Generative models beyond text
How images, video and speech are generated. Compares the main model families, builds diffusion from first principles, and explains why diffusion inference costs so differently from chat.
LLMs generate text one token at a time, from left to right. Images, video and speech use related ideas, such as likelihoods, latent spaces and transformers. The dominant image stack today is diffusion, not next-pixel prediction. This topic places the families side by side, builds diffusion from first principles, then covers latent diffusion, video and speech, and why image inference costs so differently from chat.
Engineers who only know LLM decoding often underestimate image latency ("it's just another forward pass") or overestimate the difficulty ("we need a GAN expert"). After this topic you should be able to explain how diffusion is trained, why latents help, what guidance does, and how step count trades off against quality.
Concepts in this topic
Read in this order. Each page builds on the one before it.
| Step | Concept | What it answers |
|---|---|---|
| 1 | Families of generative models | What are the main ways to model images, and what does each cost? |
| 2 | How diffusion learns to denoise | What does a diffusion model learn, and how is it trained? |
| 3 | Latent diffusion, guidance and conditioning | How do you make it fast, and how do you steer it toward a prompt? |
| 4 | Fewer steps, samplers and distillation | How do you cut the number of denoising steps? |
| 5 | Diffusion inference cost | Why does serving an image model differ from serving an LLM? |
| 6 | Speech and video generation | How do audio codecs and video models change the picture? |
Common mistakes across the topic
- Underestimating image latency. A diffusion image runs the whole network many times, not once per token.
- Judging image quality by the training loss. Loss shows that training works. It does not show that the images are good. Compare samples on your own prompts.
- Assuming LLM serving tricks transfer unchanged. Batching and caching ideas differ for denoisers. See the inference cost page.
The score-based view (optional)
Another way to read diffusion is that the network estimates the score, which is the gradient of the log density of the noisy data. Denoising score matching connects that view to noise prediction. Samplers then follow dynamics that move toward high-density regions. You can use DDPM well without this view. It helps when you read papers that use "score" or "velocity" language.
Evaluating and shipping
Image quality has several dimensions, so no single metric is enough. FID and CLIPScore-style metrics measure distribution match and prompt alignment. Human preference raters are the most trusted signal, and LLM judges are noisy. For a product, always test on your own prompt distribution, since academic benchmarks transfer poorly to a specific brand style.
Generators can produce non-consensual or deceptive content. Production systems add prompt filters, classifiers on the outputs, provenance watermarks where required, and rate limits. These are product and policy layers around the sampler. They are not part of the diffusion loss.
Key takeaways
- Diffusion and flow models lead on image and video quality today, while GANs remain in niches that need one fast pass.
- Diffusion learns to reverse a gradual noising process. The standard training objective predicts the added noise.
- Latent diffusion with classifier-free guidance made high-resolution text-to-image practical.
- Diffusion inference is multi-step and compute-bound. LLM decoding is sequential and memory-bound.
Further learning
Welch Labs and 3Blue1Brown give a visual introduction to diffusion and text-to-image models. It is the best warm-up:
Outlier's walkthrough covers the DDPM math, if you want the equations after the visuals:
Related
Modern LLM architectures
Mixture of experts, GQA and MLA, long context, state space models and multimodal transformers.
Multimodal inputs and outputs
Images, PDFs, audio and video in LLM apps: what works, what it costs and how to evaluate it.
Serving embeddings, rerankers, speech and diffusion
Inference beyond chat LLMs. Encoders, speech, diffusion and classic ML each have a different bottleneck and a different batcher.