Transformers and language models
Diffusion inference cost
Serving an image model is a different systems problem from serving an LLM. Each image runs the denoiser many times over the whole latent, so the cost is steps times denoiser cost, not one pass per token.
An LLM decodes one token at a time, and each token reads the weights and the KV cache. A diffusion model does something else. It runs the whole network over the whole latent, once per denoising step. Serving diffusion therefore needs its own arithmetic, and LLM folklore about batching and caching does not carry over unchanged.
LLM decoding versus diffusion
| Typical LLM decoding | Typical diffusion image | |
|---|---|---|
| Sequential dependency | One token after another | Many full-network steps on the whole latent |
| Bottleneck | Memory bandwidth, for weights and KV cache | Compute, for repeated dense forward passes |
| Batching | Paged attention and continuous batching | Batched UNet or DiT forward passes, step by step |
| Latency knob | Tokens, speculative decoding | Step count, distillation, resolution |
Walking through a latency estimate
Suppose one forward pass of a diffusion transformer costs F FLOPs at your latent resolution. The sampler runs 30 steps, and classifier-free guidance needs two passes per step if you run the conditional and unconditional branches separately:
30 steps × 2 passes = 60 forward passes ≈ 60 F per imageDistilled to 4 steps with the same two-pass guidance, the count drops to 4 × 2 = 8 forward passes, about 8 F. Batching the conditional and unconditional branches into one call does not reduce the FLOPs, but it can reduce the overhead.
Example
If F is comparable to a 1-billion-parameter transformer's forward pass on the latent grid, the 30-step version costs tens of LLM-sized forward passes per picture. Step count and guidance dominate image API cost in the same way that batch size and the KV cache dominate chat API cost.
Do not forget the autoencoder
Measure end to end, including the VAE encode and decode. People often attribute all the latency to the sampler and miss the cost of decoding the latent back to pixels. Profile the whole pipeline, not just the denoising loop.
Common mistakes
- Porting LLM batching unchanged. A denoiser batch is a batch of whole-image forward passes, with its own memory profile.
- Counting one forward pass per image. Multiply by the number of steps, and by two when guidance uses separate branches.
- Measuring only the sampler. The VAE decode is part of what the user waits for.
Check yourself
- With 30 steps and two passes per step, how many forward passes does one image need?
30 × 2 = 60. - Why profile the VAE as well as the sampler? Decoding the latent back to pixels is part of the wait, and profiling the sampler alone can send you to the wrong fix.
Key takeaways
- Diffusion cost is roughly steps times denoiser cost, times the guidance passes.
- Diffusion is compute-bound and repeated, while LLM decoding is sequential and memory-bound.
- Step count, distillation and resolution are the main latency levers.
- Measure the whole pipeline, including encode and decode.
Further learning
The serving other models page covers how these workloads are run in production.
Related
Serving embeddings, rerankers, speech and diffusion
Inference beyond chat LLMs. Encoders, speech, diffusion and classic ML each have a different bottleneck and a different batcher.
The KV cache
What keys and values the cache stores, how big it gets per token, and how GQA, MLA, quantization and offloading shrink it.
Fewer steps, samplers and distillation
Textbook diffusion needs hundreds of denoising steps. Better samplers, noise schedules and distilled models produce good images in a few steps, which makes interactive products practical.