Transformers and language models
Families of generative models
Autoregressive models, VAEs, GANs, diffusion and flow matching all learn to sample new data. They differ in how they generate, and in what they make easy and what they make hard.
A generative model learns a probability distribution over data, such as images, and can draw new samples from it. Several families do this, and each makes a different trade-off between sample quality, speed and training stability. Knowing the trade-offs tells you which family to reach for.
Concept
Training fits the distribution to the data. Inference draws new samples from it. For a text-conditioned image model, the distribution is p(x | prompt): images conditioned on a prompt.
The five families
| Family | How it generates | Strength | Weakness |
|---|---|---|---|
| Autoregressive | Predicts each token or pixel from the ones before it | Clear likelihoods, controllable decoding | Sequential, and slow for large grids |
| VAE | Encodes to a latent, then decodes; trained on a bound on the likelihood | Fast decoding, useful latents | Blurry samples when the decoder is weak |
| GAN | A generator and a discriminator compete | Sharp images in one fast pass | Unstable training and mode collapse |
| Diffusion | Learns to reverse a gradual noising process | High quality, stable training | Many network calls per sample |
| Flow matching | Learns a velocity field along a path from noise to data | Can need fewer steps | Tooling is still maturing next to diffusion |
As of 2025 and 2026, latent diffusion with transformer backbones powers most open image and video systems. Autoregressive models compete in audio and some image tokenizers. GANs remain in niches.
Autoregressive images
Before diffusion dominated, and still in some systems, an image is turned into a sequence of codebook IDs by a VQ-VAE. A transformer then trains with next-token loss, exactly as in language. The upside is one unified language-model stack and clean likelihoods. The downside is long sequences, often thousands of tokens per image, and artifacts from the raster order.
Diffusion won open image generation on quality per unit of engineering effort. Autoregressive image models are strongest when you already have a large multimodal language model.
VAEs in one paragraph
A VAE encodes an input to a distribution over a latent z, samples z, and decodes it back. Training maximizes a bound on the likelihood, made of a reconstruction term minus a KL term that keeps the latents close to a prior. A weak decoder gives soft, blurry images. A VAE is still valuable as the autoencoder that compresses images for latent diffusion.
GANs in one paragraph
A GAN trains a generator to fool a discriminator that tries to tell real samples from fake ones. In theory, the generator ends up matching the data distribution. In practice, the two networks can oscillate, and the generator may collapse onto a few modes. GANs still appear in super-resolution and in real-time avatars, where a single fast pass matters most.
Common mistakes
- Choosing one family for everything. The best family depends on whether you need speed, sharpness, likelihoods or stable training.
- Picking a GAN for stability. GAN training is the hardest of these to keep stable.
- Treating image-token models like LLM decoding. Each image is a long sequence, and its cost profile differs.
Check yourself
- Which family tends to give blurry samples when its decoder is weak? The VAE.
- Why are GANs hard to train? Two networks compete, so training can oscillate, and the generator can collapse onto a few modes.
Key takeaways
- The main families are autoregressive, VAE, GAN, diffusion and flow matching.
- Each trades sample quality, speed and stability differently.
- Diffusion leads on image and video quality today, and VAEs still supply the latent spaces.
- Autoregressive image models shine when they share a stack with a language model.
Further learning
The next page builds diffusion step by step, from the noising process to the training loss: how diffusion learns to denoise.
Related
Generative models beyond text
How images, video and speech are generated. Compares the main model families, builds diffusion from first principles, and explains why diffusion inference costs so differently from chat.
Modern LLM architectures
Mixture of experts, GQA and MLA, long context, state space models and multimodal transformers.
Multimodal inputs and outputs
Images, PDFs, audio and video in LLM apps: what works, what it costs and how to evaluate it.